[gregmac]

I can think of a couple challenges. Let's say you're debugging "the website won't load."

Often debugging is a hunch, and quickly branches. One person will go down the path of trying the same site from another PC. Another will try a different site from the same browser/PC. Another will try to ping the server.

If you're running this live you're always going to have the chance where someone's hunch or first try basically pinpoints the problem and eliminates a dozen checks from their consideration. If you're looking for things like "did they check DNS? Did they try another site?" but they skipped that and immediately figured out apache wasn't running, do they lose points? Or do they get 100% for their lucky (and/or experienced) guess?

If you do it on paper, the major problem is basically infinite branching for every step and possible result. How do you mark that? What's the minimum you need to do to get 100%? I think it's also unfair, because even as a seasoned vet I've been deep in troubleshooting to the point of (for example) greping source of nginx for a specific error message -- which I'd never have even thought of two steps earlier. I do t expect anything that complex on a school test of course, but the point is the result of each step is often the only way to even think of the next.

[atoav]

Sure testing a beginner is hard here. But testing someone who learned how to debug means testing their knowledge of the moving parts of the system they are debugging. And this can be explained by the person being tested.

E.g. "I am going to ping an IP on the outside first, to see if ICMP message reaches the outside server. If yes I will check if the DNS server responds, if no we check the physical connection."

That would be an ideal answer, judging less ideal answers fairly is certainly a challenge, but not impossible.

[no1lives4ever]

And what if the firewall blocks icmp? What if the dns server is internal to the network and is returning a stale ip? There are way too paths down this rabbit hole.

[gregmac]

This is exactly my point. It's even worse in a "school test" situation, because who knows what contrived scenario the instructor has invented depending on what their focus is (eg: DNS vs physical net vs routing vs apache config).

Maybe there are multiple DNS servers for the zone that are returning different IPs, maybe one or both of them is even in split-zone configuration so it returns a different IP depending on if you're internal or external. Maybe the client has manual DNS configured or a hosts entry that's wrong.

Each of these problems would have several more layers of troubleshooting steps and branching, and it's not even a complete list -- and this is only if the problem is DNS-centric! There's hundreds of other branches for each of the other problem categories.

[atoav]

I mean if it is a test you obviously test things that you tought them first, or not?

Of course in reality there can be more, weirder things — especially if you are coming into an unknown network. But we are talking about an educational context here, would not make a lot of sense to let your students run into new unknown issues on a test unless your goal is not to educate.

[sideshowb]

More simply you could just ask to brainstorm 5 things the issue might be.