Everyone thinks they’re going to give feedback because “we’re different, we’re going to do this right”. Then life happens and that shit gets dropped by the wayside.
We don't really give individual feedback on the take home challenges. Where we struggle is explaining this to people, it's not obvious to every dev _why_ we don't give specific feedback. Here's what we send:
---
Unfortunately, we’re not going to be moving forward with your application for the backend role at this time.
The first thing we want you to know is that we got a lot of applications for these roles. Like, a lot a lot.
The way we evaluate candidate submissions is that we’ve built, over a year or so of running this challenge, a written rubric for what the strongest submissions look like. We started with something sane looking, and then iterated over time as we got a sense of what candidate submissions actually looked like.
We score submissions according to that rubric. Different members of our platform team score different applications; they’re just jumping in and looking at the code and making grading decisions. In the pool of candidates we’re evaluating right now, your code submission missed our cutoff.
A natural thing to want from us next is a copy of the rubric, or specific information about what your submission was missing. We’d want that too! But we can’t give that to you, for (at least) two reasons:
(1) The outcome on your code was a weighted combination of a bunch of different factors, so there isn’t a simple answer that doesn’t just dump our whole rubric to you.
(2) We want to preserve the opportunity for people to apply to these roles in the future, and spoiling the challenge would break that.
What we can tell you is that the most successful submissions had a combination of these attributes:
- Heavy user focus. Successful submissions explained how the system could do good things for users.
- Straightforward database models that make the queries we care about fast to run.
- Transactionally safe sync with Stripe. For example, the best submissions mentioned that Stripe API requests had to be idempotent to ensure the user is not charged multiple times.
We sincerely hope you stay in touch and re-apply in the future. In the meantime, we'd like to give you some Fly.io credits to play around with! If your email address here matches your Fly.io email, click here to get them. If not, please let us know which email you use for Fly.io and we'll set that right up.
Please feel free to reach back out to us about other roles, or about this role in the future. Thank you, again, for doing this.
Seems like you probably are not explicitly stating this policy beforehand? It's not very surprising that explaining why you don't do as an applicant might expect afterwards is not going well. From the Github examples you've linked earlier, right at the end:
> When you’re ready, let us know and we’ll schedule it for review. We review submissions once a week. You’ll hear back from us no matter what by the end of the /following/ week, possibly sooner if you submit early in the week.
+ 1 sentence expressing "we will tell you if you passed or failed, we have a policy of not providing actionable feedback"
> Remember, we are not timing you and there is no deadline. Work at your own pace. Reach out if you're stuck or have any questions. We want you to succeed and are here to help!
+ 1 sentence "but make sure to not spend significantly more time than X hours"
I think that your heart is in the right place with a message like this, but my feedback would be to keep it shorter, perhaps to a paragraph or two. There is a phrase called “the message is the medium” and I think it applies to hiring.
The best way to reject a candidate IMO is to call the candidate, deliver short feedback over the phone “Some of the feedback we got from the interviewers was we like to see the problem solved in a shorter amount of time/ less space complexity/ whatever). If I was on the fence, I would them know that if they want to try again they can give it another shot in 6 months, a year, whatever. To me this seems like the most humane way to do it (This was actually my experience with Google, maybe 8-10 years ago).
At the core of everything people-related a company does or does not do is the fear of a lawsuit, and in the case of interviewees, the only thing that leads to a lawsuit is telling the interviewee anything at all - therefore, no feedback.
No, replace "sued" with "humiliated". Every time. It just sounds more prudent to say "legal risk" than "we're worried a member of staff will make us look like absolute morons" or "we're lazy". It's obvious when you think about it.
"We didnt hire her because she seemed like she might get pregnant" isnt the kind of feedback most people write anyway and if they did a 2 minute pass via HR is enough to eliminate the risk.
The companies that are actually worried take pains to tell you what not to say during the interview. This is rarer than you'd think. Most companies aren't actually that worried about getting sued over interviews. That doesnt stop them from tossing this lame excuse at you though.
I agree with this, a LOT of programming assignments have numerous effective approaches to a solution, and it's quite often, I believe, that the interviewers are simply unfamiliar with the submitted solution and don't know how to evaluate it or what to look for, not that it doesn't work.
And of course, this is more likely when the interviewee submits a more advanced or higher-level solution, meaning the best strategy is to "dumb-down" the submission rather than trying to use the latest language features.
---
Unfortunately, we’re not going to be moving forward with your application for the backend role at this time.
The first thing we want you to know is that we got a lot of applications for these roles. Like, a lot a lot.
The way we evaluate candidate submissions is that we’ve built, over a year or so of running this challenge, a written rubric for what the strongest submissions look like. We started with something sane looking, and then iterated over time as we got a sense of what candidate submissions actually looked like.
We score submissions according to that rubric. Different members of our platform team score different applications; they’re just jumping in and looking at the code and making grading decisions. In the pool of candidates we’re evaluating right now, your code submission missed our cutoff.
A natural thing to want from us next is a copy of the rubric, or specific information about what your submission was missing. We’d want that too! But we can’t give that to you, for (at least) two reasons:
(1) The outcome on your code was a weighted combination of a bunch of different factors, so there isn’t a simple answer that doesn’t just dump our whole rubric to you.
(2) We want to preserve the opportunity for people to apply to these roles in the future, and spoiling the challenge would break that.
What we can tell you is that the most successful submissions had a combination of these attributes:
- Heavy user focus. Successful submissions explained how the system could do good things for users. - Straightforward database models that make the queries we care about fast to run. - Transactionally safe sync with Stripe. For example, the best submissions mentioned that Stripe API requests had to be idempotent to ensure the user is not charged multiple times.
We sincerely hope you stay in touch and re-apply in the future. In the meantime, we'd like to give you some Fly.io credits to play around with! If your email address here matches your Fly.io email, click here to get them. If not, please let us know which email you use for Fly.io and we'll set that right up.
Please feel free to reach back out to us about other roles, or about this role in the future. Thank you, again, for doing this.