[lxe]

How do you calibrate yourself on these open-ended questions? How can you compare one good answer to another completely different good answer and figure out which one is better overall?

You can't rely on a "The very best engineers have a checklist they apply depending on the type of bug." type of framework here -- each "very best" engineer might have a completely different way of doing things.

You have to have at least some questions that are difficult to answer but have a countable range of responses, and watch for specific lapses that can show how well a candidate demonstrates specific experience required in your job description.

For example, if you require some experience in distributed systems or scaling things to many users, have them talk about "Could you talk about an or program you built to scale to n users, what performance or reliability issues did you see (or foresee), and how did you solve them?" Have a loose set of industry specific things to watch out for, like caches, load balancers, monitoring/observability, etc.... You don't have to have the specific knowledge yourself, but if the candidate can explain to you what they did, and how/why, in the framework of the specific experience you're looking for, that's good signal!

[hinkley]

> The very best engineers have a checklist they apply depending on the type of bug.

This is misleading at best.

Masters do less work than beginners. They have years of intuition built up that let them apply heuristics extremely cheaply and save their energy for solving the novel parts of the problem. There is no 'list'. There is only the verbal part of the brain trying to introspect on what just happened. Sometimes accurately, but quite often not. The ones who get it even close to right become mentors, instructors.

I can spend as long as you like explaining my debugging process and the first time we sit in a war room you'll call me out for not following my own 'list'. Every situation is either unique or the result of a lack of self-reflection (why do we keep solving the same problem over and over? Are we not programmers?)

The biggest danger to people who can't explain at all is when advances in their field arrive and they can't introspect enough to adjust their behavior. There are limits to how much you need to address this in an interview process. For many of these people, they'll already be in that space when you meet them and easy enough to spot.

[rubidium]

Most problems aren’t special. They occur because some good general guidance wasn’t formed or applied.

Having a checklist does help masters. It helps make sure they don’t miss the obvious issue in front of their face.

Checklist manifesto is good reading on this.

[hinkley]

Hmmm. I think the checklist exists, for sure, but tends to be more Plan B. If your initial strategy is exhausted with no progress, fall back on the checklist.

As a programmer I have an opportunity not everyone else has. I can codify parts of my checklist in a way where I don't really have to think about them at all save for a few times a year with the big outlier problems, teaching someone to do basic troubleshooting themselves, or starting a new project.