[garyjlin]

We're mid-way through building a V2 of the scorecard. This "Senior - Weak" grading model isn't working, since everyone has different ideas of what a senior engineer is.

Instead, we're going to just expose raw data about somebody's code submission, and let you decide what those numbers mean.

Some examples:

1. 7/10 unit tests passed

2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB

3. Linter returned 10 warnings, 0 errors

4. Cyclomatic complexity counter returned max of 24

5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars

[yunohn]

It's good to hear you're working on this, it's a hard problem to solve for sure.

In my experience, the metrics you've sampled aren't particularly useful for hiring. Some of my thoughts:

> 1. 7/10 unit tests passed

Automated unit tests are table stakes on all platforms like HackerRank, LeetCode, etc. Might be useful to classify/clarify the failing ones? Or even add an option for the candidate to clarify their thought process for those.

> 2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB

As an engineer, I see the appeal of OOM checking. But as a candidate, it would be important to have the same information available during test-time, so I could fix it. Could be as simple as an accidental N+1 ORM mistake. Otherwise, hidden performance testing would make the experience very stressful, as then I need to double-check everything and over-optimise for perf.

> 3. Linter returned 10 warnings, 0 errors

I would argue that linting should be out-of-the-box and automated for the candidate. Or even if not, there should be heavy filtering of linter warnings. Every company has their own style guides, and candidates shouldn't be expected to learn that for an interview.

> 4. Cyclomatic complexity counter returned max of 24

While I do understand the concept of Cyclomatic complexity, I have /never/ seen this used IRL in any company (startup, mid-size, FAANG) that I've worked at. I would much rather judge code complexity in terms of readability and decision making.

> 5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars

I appreciate clean git commits, but these are enforced by company's internal policies during code reviews. Easy to learn once, and continue doing. But definitely not a deal-breaker in a time-constrained interview assignment.

Are these raw metrics you shared what constituted the grading from V1? I would definitely reevaluate most of these for V2, as they seem a little nitpicky than insightful.

[ajacksified]

Not only that, but this is an interview question. It isn't helpful at all to judge the quality of code in isolation; nobody's going to write perfect code in an interview setting, especially one in which the candidate can't have a conversation with the interviewer. When I conduct code interviews with candidates, we pair on projects. The candidate is asked to complete the task without regards to how clean the code is. We then have a conversation afterwards about how to clean things up, solve for complex cases, etc. It's much better signal to talk through these things to get the candidate's insight than it is to just get a chunk of code at the end that may-or-may-not meet the linting guidelines of my company.

I've interviewed hundreds of engineers in the last twenty years. I've been (and am) a hiring manager for many of them. I wouldn't touch this with a ten foot pole, and I'd run away as a candidate if my git commits were being judged as part of a coding interview.

These "metrics" are for conversation, not for automated "judgement."