[danpalmer]

I've seen those with AWS certifications be unable to put anything together on AWS and I've seen exceptional engineers with plenty of production AWS experience fail the certifications.

Security experts fail to get CISSP qualifications, while those who can barely use a computer pass. The security industry considers CISSP to be a bit of a joke for this reason.

This sort of certification is in my experience, sought out by those who don't have much else to show, and therefore is often correlated with not being very good. You can learn by rote and pass with no understanding to tick an easy box.

I think it's a shame GitHub is getting into this, but cynically it makes sense for them.

[3pt14159]

I personally dislike tests that are multiple choice. It loads the mind with (sometimes obviously) false data that must be rejected and the ability to pattern match on word usage or even guessing between answers that haven't been eliminated means that the instructor never really knows if the person actually knew the answer and neither does the student.

When I used to participate in math contests I'd go out of my way not to read the answers before attempting to solve the problem because it's much more gratifying that to reverse engineer the problem from the input. In the end, I'd resort to that or guessing if I couldn't crack the problem on my own, but then at least I knew which problems I really solved and which ones I didn't. And when I was invited to do more complex math contests than the ones that Waterloo was putting out I was astonished at how much harder just having to write in an exact answer was, since you missed out on the "not one of the options" feedback if you flubbed a number somewhere.

With these certification tests it's impossible to really do the same thing. It doesn't even feel like real thinking compared to writing software or prose. Also the pointless memorization of things like "what classes of MITM attacks are there?"[0] don't really get a person to learn the stuff. It's like memorizing the atomic weight of elements. That stuff never stuck for me until I was doing real chemistry and the basic act of looking up a weight each time slowly got me to memorize the elemental weight of the chemicals I was working on. Same goes for all those French words I was forced to learn in Canada. Once I started wanted to speak Russian I was amazed at my ability to actually be able to remember words.

I'm not saying these things should never be tested. I wouldn't trust a structural engineer that didn't know what lateral torsional buckling is[1] but the manner in which we test it should be as close the reality we would expect, and this is what we do in engineering school most of the time.

[0] Answer: Surveil, Fabricate, Modify, and Deny.

[1] Essentially the concept that a tall but thin beam can swing to the side, or "torsionally buckle" when loaded from the top (thus requiring bracing) unlike a beam rotated 90 degrees, which does not require bracing for this reason.

[danpalmer]

You're completely right, the testing method is likely very linked to why this is an ineffective way of deciding someone's qualification.

I think multiple choice exams can be a good way – a physics exam I took was multiple choice, and so because the examiners were "giving you the answers", they took it as an opportunity to ask really out-there questions. They would frequently take two wildly different aspects of the physics course and mash them up in a new way. I thought this was very effective at testing if the student understood the areas and could therefore combine understanding with intuitive leaps, or whether they had memorised the formulas necessary for the exam and could only repeat those on command in the exact form they knew.

This suggests to me that it's less multiple-choice, and more that this is a purely fact-based style of testing.

For a computer security course I took at university one of the exam questions was "Describe Stuxnet – 20 marks" (half the exam's marks). we had had a lecture dissecting the whole Stuxnet incident. For those simply memorising facts this question would be quite hard, but for those who had understood why certain things matter and could write an in-depth explanation of the security failings, it was great.

The problem is that marking this sort of question requires a significant amount of manual work, and that doesn't scale. Another example would be Phd vivas, which I've heard are generally a well respected way of determining ability, but which again take a significant amount of expert manual input.

I don't think we'll get good certifications for these sorts of things until we find better ways to examine like this.

[curryst]

Long form answers (and college courses in general) are graded relative to your peers, whereas certificates aren't, which creates a huge difference. Even long form answers that aren't actually bell curved at the end, the person grading it has still read the best answers and the worst answers, and they're probably going to grade relative to that.

That precludes having studied "enough". "Enough" is measured relative to your peers, not to how hard the exam is. As long as someone else is studying, "enough" is a moving target. On a certification, I can check out how hard the exam is and study just enough to pass it. If I know for a fact that the exam isn't going to test comprehension, just recall, and that I'm not going to fail because someone else studied enough to comprehend, it makes sense to just memorize the recall portions.

Certifications would do better if they went the same way. For each group of people taking the test, curve the test so that some minimum, fixed percentage of them fail. Let's say it's 20%; so for every group of 100 people taking the test, at least 20 of them will fail (although it could be more if you set a floor on the lowest score you will allow to pass). Even if the exam doesn't test comprehension, I'm willing to bet that most people would pick it up just through studying.

[7ewis]

I've done one of the AWS certs a few years ago, found it was very much a memory test.

A lot of questions on easily Google-able things. I learned a lot from the training material, but don't think the exam itself really proves that much skill on the platform.

If I took the test again now, I doubt I would pass despite a lot more experience since I passed the first time around.