[e4e78a06]

Curious how this will work with more complex cases? E.g. distributed systems, concurrency safety, etc. Also, how do you deal with solutions that don't compile but are 95% of the way there conceptually? I find that's a common occurrence in non-FAANG settings that generally would still mean a pass to the next round (sometimes even with strong perf ratings).

And how do you prevent someone from cheating if there's no engineer there on the interview?

[garyjlin]

Great set of questions!

1. Complex cases:

We can spin up isolated AWS or GCP environments with some resources pre-filled, and the rest is up to the candidate to build. Basically a systems design whiteboard challenge, except instead of a whiteboard, it's actually in a cloud environment. Since "input" will be defined, and specific "output" will be expected, we can then run tests against this environment to test for sufficient robustness. For example, if the prompt is to build a load balanced CRUD api, we can run create resource a million times, then try to read, update, or delete, and measure total response time. For something like testing concurrency safety, we can run a bunch of parallel threads attempting to access the same resource, then test for timeouts (deadlocks) and inconsistencies.

2. A solution that's code complete but has a minor bug that prevents compilation

For a solution like this, code completeness scores are unfortunately going to be 0, simply because there's no way to test for functionality, performance, event logs, if it doesn't compile. However, our static code analysis will still work, so they may still get high scores on code style, code complexity, naming, etc. Something we might be able to add is a way to flag to employers that a submission has high scores for everything except a specific metric, and so we'd recommend manually diving into the code.

3. Cheating

We can't prevent cheating if the candidate gets their roommate to do the interview for them. However, if someone just blatantly copies an old solution their friend sent or they found on the internet, we can check for that. We can run a diff between the submitted git history and all previous submission's git histories. 2 legitimate final solutions might be close (ie 60-70% match), but you'll never have 2 organic git histories with multiple commits be close, (it'll pretty much be either 0% or 100% match). Looking things up on the internet isn't considered cheating, I don't know of a single dev that could survive the day without SO :D

[phibz]

> unfortunately going to be 0, simply because there's no way to test for functionality, performance, event logs, if it doesn't compile.

You could always have someone look at their work. Or better yet walk through it with them to try and ascertain their thinking. This is what I do with my team, multiple times week, in my actual job. Seems better than immediately excluding them

[onlyrealcuzzo]

If TC can run the code (not sure why not in this environment) - then it not compiling should probably be a no hire.

[e4e78a06]

Strongly disagree if we are talking a complex problem like coding a distributed system. It is very possible for an implementation to be 90% correct but it won't compile or it's wrong for very subtle reasons. And maybe if there was another 30 minutes it would be debugged correctly but it didn't work out in the 45 minute interview period. There is signal contained in the 90% right attempt that can show a lot of knowledge and skill which you'd be missing out using autorejects.

We are not talking some CRUD app where it is trivial to make each component work or a Leetcode problem that you can spit out in 20 minutes.

[ytpete]

I'd still argue it's a bad sign if the candidate is stuck with nothing even compiling after 45 minutes. That's not a sign of the problem being complicated, that's a sign of the developer either (a) struggling to figure out dev environment basics, or (b) having a tendency to go way down one solution path without pausing for any simple gut checks / sanity tests of their work in progress.

In the latter case, is it possible for a developer to operate that way and still produce a great solution given more time? I suppose so - but in my experience, the people who are not in the habit of incrementally checking their work are virtually never strong developers.

There's also a time management question here. If a person knows they only have 45 minutes, that should create extra pressure to start testing earlier. Someone who still waits until time's almost up before checking anything is probably a fairly extreme case of this antipattern.

[garyjlin]

Yeah this is actually why we recommend liberally-timed take homes instead of a sit down 45 minute block. It's super frustrating to get 95% of the way there just to be cut out by time.

[aronowb14]

love this question: especially about the distributed systems.

Something I've been thinking about, not necessarily in the interviewing world, would be to have a simulation of distributed systems for non-production code: in theory I think you totally could have a simulation layer for databases / server loads and test how your code performs in such environments.

I guess it exists in production: but haven't seen it for either a) learninga bout distributed systems, or b) for interviews