[cspa]

Excellent points -- this is something we discuss extensively. After talking with so many employers, we found that each of them look for something different.

Adaptive scoring algorithms and ways to let employers create custom projections of our multidimensional data is something we're thinking about. If you have ideas on how to build this, I'd love to hear them! [e-mail]

[ABCLAW]

Hey James.

The issue you're having is that you're trying to test over 7 dimensions, with the majority of those dimensions being irrelevant to an individual taking the test to signal mastery of 2-3 of them.

The standard method in testing to deal with issues of this nature is to provide people with the opportunity to test themselves on modules, rather than taking the omnibus test so that they can spend the majority of it signalling that they aren't good at things they don't care about. If they're good at everything, they can sit for all of the modules. If they aren't, they aren't wasting their time.

Additionally, this frees candidate time up to let you test people in more depth in the areas they're actually professing skill in. From that point, you're open to do interesting things; evaluate question difficulty/time to completion and adjust the questions provided as the test unfolds to match the expected skill of the applicant for instance.

As it is now, you're getting buy-in from corps that little have no skin in the game, but I don't see how the sample test you have provides more signalling value than an applicant having a CS degree or a list of github projects they can discuss.

[cspa]

Re: custom modules. One (arguably) interesting thing we do is to only take the MAXIMUM subscore of all 7 topics and use that component in the composite score, ignoring the other 6 topics. This allows candidates to focus on only certain topics if they want. Employers want someone who is strong in one area, rather than a jack-of-all trades and master of none.

Adaptive testing (https://en.wikipedia.org/wiki/Computerized_adaptive_testing) is something we're looking into, and would be extremely interesting (and challenging!) to build.

[ABCLAW]

Hi James,

The algo to take Core + the best other section, right? As it stands, the test difficulty makes it pretty easy to hit 2 perfect scores, so you'll be competing against those pretty consistently.

How does the CSPA help my interviewing process if I'm a network guru and knock all those questions out of the park, but apply for a ML job where my knowledge is paltry? How does my CSPA testing help me differentiate from the above guy as a network guru when I also aced the ML section? This seems like an obvious area the test can outperform other testing metrics.

As far as I can tell, the only value for the current scoring system is to weed out people who are just plain abysmal at everything. Eventually people are going to game your exam and pop up very close analogues onto google. At that point even the floor function is dead with the added deadweight cost of the exam still being an application requirement at BigCo.

Maybe I'm just blinded because I think having accurate skill radar charts that test takers and employers could use for self improvement and prospect evaluation, respectively, could be an absurdly large value add.

[cspa]

Yep I agree! In fact, about 50% of the people who take the CSPA, do it more for self-evaluation than for employment purposes.

You're right, so far most of the test takers are entry-level or changing careers. To accurately assess specialities like ops/IT, we'll need separate, dedicated subject tests (or adaptive tests).

That said, no one has yet gotten a perfect 1600 :) . The highest is something like 1380.

[ABCLAW]

Cool! Thanks for the clarifications. I'm looking forward to whatever comes next!