[shikharja]

ska, our aim with the challenge was to allow candidates to not be biased by a fixed outcome and try to solve the problem as they would solve any real data science problem.

This meant we couldn't automatically score/rank a candidate's solution. We do provide them with an evaluation metric in the problem description (Mean Absolute Error). Here is scoring rubric we provide to the interviewers when they review the submission - https://d.pr/i/hNYY0u

Would love to hear more opinions on our scoring rubric

[ska]

That's useful information, thanks. I'll add some thoughts on the rubric.

[edit]. Initial thoughts:

- "data wrangling" scoring difficult given this task - more weight to "rationale", that's more important the "performance", here.

- not enough focus on communication capabilities

- really need something on validation

- "proficiency" measure you use is pretty much impossible to accurately evaluate from your example question

- way too much weight to modeling section overall

[shikharja]

By 3rd bullet, you mean Data Validation?

[ska]

both data validation and model validation are important.

Of course more abstractly you want to validate the entire process - but I was referring to those two, as it is hard to see how to address the latter in a format like this.