[viraptor]

> Often it's less what you know, or could do; and more about "can you do the test?"

It really depends on how you create the test and how you review it. You can prepare one where the actual solution is 10 lines of code and anything else is exactly for showing what the candidate knows / could do.

I used something like "read data from CSV, write it to sqlite, treat it like a mature production app, feel free to use placeholders (usage doc goes here), go nuts". Then the test was really about knowing about error recovery, encoding issues, documentation, error reporting, monitoring, ci pipelines, etc. Many badly organised tests don't make the whole concept of a take home test bad.

[frumper]

Your test sounds like what people complain about. You give them instructions but it's "really about ... ". Just ask them what you want them to know. If you hired this person and they turned in code without error recovery or encoding, you'd tell them they need to add it and they would. No need to obfuscate your intentions and have them read your mind in what you're looking for.

No take home test is even needed. A simple conversation would let you know if they understand the importance of those things.

[viraptor]

> Just ask them what you want them to know.

You can't really ask about those things directly. If I ask "(how) would you add monitoring to this?" you can make something up on the spot. If I ask "what else would a mature production app contain?" that will give me a better idea of what you're familiar with.

> you'd tell them they need to add it and they would

Different levels. We were not after juniors in that case, but people who can do independent work. The question would be way more specific otherwise.

> No need to obfuscate your intentions and have them read your mind in what you're looking for.

It's not about mind reading. For that role, you either know how to create/manage serious applications or not. Your specific ideas may be different than mine and that's fine, but the general areas of interest will match. E.g. whether you register some event notifications, do text logging, persist decision journal, or do something else — you'll likely mention that. And you may forget/miss one or two things - that happens. But you don't miss them all if you're a good candidate for that position.

[frumper]

When you say make it up on the spot, I'm hearing they can answer questions when you ask. If you ask them to describe how they'd process input and send it to a database they can just as easily tell you what they generally do. If they don't mention any of the things you're looking for, then you have your answer. If they answer some then you can decide to probe about what's missing, or thank them for their time. You haven't explained why a take home test is needed for that.

[viraptor]

> When you say make it up on the spot, I'm hearing they can answer questions when you ask.

Or they have enough surface knowledge to risk an answer that could potentially work. It depends if that's the threshold acceptable to you.

> You haven't explained why a take home test is needed for that.

We're commenting on the blog post about this very question. Making the initial stage of this test take-home is at least partially solving points 2,5,6,8 from the list.

[frumper]

It sounds like we're now back to mind reading the answer you'd like them to provide based on overly broad questions. That is a poor way to assess whether they know about a topic. Ask them what you'd like to know. You don't have a relationship with them and they aren't familiar with your version of "normal" or "mature" processes.

[viraptor]

It doesn't matter if they're familiar with my specific concept of a mature app. (Although just reading the job description would cover 90% of it...) There's always going to be a lot of overlap of concepts anyway. The question was broad exactly because there are many good answers.

(FTR, the answers were pretty consistent in what elements they included, so it wasn't confusing for the candidates)

[emodendroket]

That's like a trick question given that interview questions typically want the opposite.

[viraptor]

The instructions were longer and clear - nobody tried to second-guess them. So not an issue in practice.