[bennyelv]

A perspective from the other side of the desk (playing devil's advocate here):

There's a fundamental skill that a good programmer has to have, and that is to be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way.

There are plenty of programmers who fake their way through a career without having that skill. They just copy stuff and never really understand it. They're fine for some types of programming career, but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.

A contrived live coding exercise gives you a strong signal on this. It does have a decent chance of producing a false negative, but only a very small chance of a false positive, and that's the trade that has to be made with this kind of approach.

Is a better option to not do this kind of assessment, hire the person, find out that they can't do the hard bits and then fire them within 6 months? I'll take some convincing of that...

[pavlov]

> “take a novel problem that they haven't seen before and break it down”

But who does that professionally as a stand-up performance, clock ticking, a judge breathing down your neck who has been equipped with a script that tells him things like “if candidate doesn’t do X within the first ten minutes it’s very bad”?

Doing well in that situation depends more on social performance skills than problem-solving skills. You’re essentially trying to infer the interview script that the interviewer has in mind and act it out convincingly.

If someone says “I’d like to start by taking ten minutes alone in another room and think it out first”, will they get hired? Probably not. It’s not the expected performance even if the solution is fine.

[kps]

Yes. I did this type of interview for a previous job, and don't want to do it again. I'm a software developer, not a salesman.

Requiring salesmanship to enter a company may well correlate with requiring salesmanship to advance and be recognized at the company. (I don't regret that job, because the rank-and-file were great, but I don't want the same structure of overlords.)

This time around, I am fortunate to have effectively lifelong runway, so I'm now trying to handle an interview as if I were discussing a problem with a co-worker. If that filters me out, that's probably best for me, too.

[jvanderbot]

Given two candidates who can break down problems, I'd prefer to hire the one who can do it with a gun to their head (metaphorically, of course). Lord knows owners, ceos, board members, managers, and angry co workers would often enough like to put a gun to my head.

[throwaway5959]

I’d suggest not working for people who like to put a gun to your head, euphemism or not.

[adave]

What's wrong with hiring for interviewers to be this nihlistic.

[jvanderbot]

Working well under pressure is a skill too.

[gymbeaux]

Why? We don’t have animals hunting us and a constant need to fend for food and water like our ancestors did. Why in 2023 should anyone be under pressure for anything to do with their job?

[jvanderbot]

Ideally, there would be no pressure involved. But in reality, there are pressures associated with deliveries and conflicting goals. You don't have to look far for these, and hiring for the ideal conditions is unrealistic.

[reidjs]

That’s why he says this approach is more likely to produce false negatives than false positives

[gymbeaux]

False negatives are always better than false positives. Every company who does leetcode interviews agrees with that.

[908B64B197]

> If someone says “I’d like to start by taking ten minutes alone in another room and think it out first”, will they get hired? Probably not. It’s not the expected performance even if the solution is fine.

When interviewing I always start with a discussion about the problem. On the whiteboard. I'm explicit with the candidate that this is totally expected.

I'm actually measuring something perhaps more important than the code that will get written later: can this person formulate a plan and have a technical discussion.

[gymbeaux]

But what you don’t know is whether people who didn’t give a stellar performance would have gone on to be adequate or even excellent hires.

[hnfong]

As for the time limit, it's relative. All things equal, if you can complete the problem in 10 minutes and the company can hire somebody that can complete it in 9, then objectively the other candidate is the stronger one.

As for nervousness for being forced to partaking in a stand-up performance, I'd argue that "social performance skills" can work against you, since the more "antisocial" you are, the more you can ignore (or are oblivious to) what other people think about you and you can focus on your task. It's only people who actually have the minimum requisite "social skills" that would be conscious of other people intensely watching them "perform".

But even if you're right -- interviewing is inherently a "social performance" activity. If you're not coding you're doing some other stand up performance to present yourself anyway.

[oceanplexian]

> All things equal, if you can complete the problem in 10 minutes and the company can hire somebody that can complete it in 9, then objectively the other candidate is the stronger one.

They are stronger in the specific problem space you’re testing for. If your business has a lot of novel coding problems that need to be solved in under 9 minutes, then sure, this is a great measurement.

On the other hand, if your business has a lot of hard problems that take days, weeks, or quarters to solve, measuring someone’s ability to solve a 9 minute problem is a terrible metric. If anything it’s a counter signal, since it tends to select for candidates that optimize short term thinking over long term planning and problem solving.

[hnfong]

> if your business has a lot of hard problems that take days, weeks, or quarters to solve

I suppose your ideal interview as an interviewer would be to give the candidate a take home task and ask them to spend 2 weeks to work on it?

> If anything it’s a counter signal

Or perhaps give the candidate a task that normally takes 30 minutes, and hire the candidate if they take 60 mins to finish it?

I mean, you do you, but I hope you (and everyone else) can see why I'm not convinced otherwise.

[frumper]

No, the ideal interview is to ask them questions about what type of problems they've solved before and ask them to walk through what they did. Also, have a conversation with the person to get to know a little bit about what they're looking for in a job/company. It's an interview, not a tryout.

Metrics like a 10 minute task, or a 30 minute task is all relative. Do they know they language, the IDE, the operating system, the documentation, any experience in whatever abstract problem topic you choose, and personal comfort levels will all come into play.

If you want to filter someone to do a very specific thing then post to hire a contractor with the specifics of what you need. If you want a developer that can grow within your organization then pop quizzes area good way to dismiss good candidates.

[edanm]

> No, the ideal interview is to ask them questions about what type of problems they've solved before and ask them to walk through what they did.

This just doesn't work. Lots of people can bullshit very convincingly, and even if they can't produce novel solutions, can very well explain them in a matter that makes you think they for-sure know how to do it.

Hell, I can probably still prove lots of interesting things about set theory in math, as can many other people who studied math - doesn't make me the equivalent of the world-class mathematicians who actually came up with the proof.

Empirically, "talk to someone about what they did" doesn't give me hires that actually know what they're doing.

[spacemanspiff01]

If they pay me at contract rates double my current salary, and in my particular case my current IP assignment clause/the problem/ law makes it ok. Sure, pay me money to do something like what you guys need done.

Probably most useful if it's a contribution to open source.

[gymbeaux]

I don’t get why I have to prove to some strangers I can write code when I’ve been doing it for the last decade. It’s a ludicrous and broken system.

[zamnos]

Because those strangers don't know you.

Any idiot can write things on a resume and say they did things they didn't do (aka lying). You would never do such a thing, of course, but as crazy as it sounds, there are people out there who would do just that! So because there's no professional license to write code, the only way to prove to these strangers that you actually can write code is some sort of exercise where you prove it to them.

I really don't get why this is so hard to understand, either. I get that live coding in front of someone else is a crazy stressful situation - I've failed multiple interviews because I couldn't perform on demand and answer the interview question in the interview setting, when I could easily have done so after taking a proverbial shower to have a think, so I'd love to get rid of them too. But unless we all band together and start a software developers guild or something, the live coding interview is here to stay. (Though, Triplebyte, now Karat, and others did take a run at improving the process, so there's that.)

I know what I know, but you don't know what I know. It's only by communicating, in a sufficiently unfakeable way, like a 45-minute in-person interview, that one can pass or fail the unwritten "can program" shibboleth.

[dragonwriter]

You don’t get why you have to prove your ability to do a job when you want them to pay you to do that job?

[Retric]

Normal interviews are mostly a scripted performance where people largely regurgitate answers to predictable questions.

So the issue isn’t that it’s a stand up performance it’s that you need to split your attention between the performance and solving some trivial problem. Becoming really good at programming requires being able to focus on an actually difficult problem not work through a simplified example while entertaining an audience.

There might be a correlation between solving trivial problems quickly and being able to figure out a heisenbug from some multithreaded monstrosity, but it’s not strong enough for minor differences in performance to be meaningful.

[Ancapistani]

> Becoming really good at programming requires being able to focus on an actually difficult problem not work through a simplified example while entertaining an audience

There are multiple ways to "become really good at programming".

A company of any size needs people who can churn through well-defined problems quickly, and most live coding tests are relatively well-suited to selecting for that. They also need people who do other things well:

* tackle large, ill-defined problems, alone or in a small team * identify, triage, and (if necessary) solve problems as they emerge * refactor existing implementations that have outgrown their initial architecture * identify trends and estimate when they will lead to scaling issues in the future * communicate complex problems to people without the necessary context * break down complex solutions into discrete, well-defined steps that can be tracked to completion

They also need people who can accurately estimate how long all of the above will take... unfortunately, as we all know, such people do not exist.

If you're hiring someone to churn through well-defined problems, live coding interviews are likely a good fit out of the box. If you're hiring someone to reinforce a weakness in your organization's ability to do any of the other things above, live coding exercises are at best a loose framework that we're all familiar with to try to get at whether or not they have those skills.

[hnfong]

I'd even argue it's a prerequisite.

If one doesn't even know how to complete well-defined tasks reasonably quickly, I can't imagine how they would be able to break down complex tasks to these smaller, well-defined tasks, and accurately estimate the expected effort needed.

This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!

[Ancapistani]

> I'd even argue it's a prerequisite.

You can know how to do something without being able to consistently do it.

Maintaining velocity is something that I've never been very good at. I always struggled as a junior and mid-level developer with having periods of very high productivity followed by periods of very low. I've been fired for it.

On the other hand, if you measure my output on a monthly or quarter time horizon, I'm very consistently at the top of the class.

My strong suits are around helping others get through tough problems, quickly switching between levels of abstraction (including within a single conversation), understanding how a part of a process fits into the whole, and in guiding/limiting architectural discussions by managing the scope of the problem space under consideration.

This fits very well with my current "staff level" role, but was a real struggle getting to this point.

> This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!

I can see why you'd say that, but I think there's a big difference: PHB was incompetent in addition to having a different role from his team. I've found a sweet spot for myself where I can fill the parts of the "PHB role" well that actually do help the team, but I'm also more than capable of switching contexts and knocking out some tickets when there's a crunch. When I'm especially "in the flow" I work on that sort of stuff and do well. When I'm not, I run interference and keep my team from having to deal with the organizational stuff that doesn't directly help them get their jobs done.

The best functioning teams I've experienced all have one thing in common: they set people up for success by allowing them to do play to their strengths. As a team grows, you can find gaps in those strengths. I see hiring as an opportunity to find someone who is strong in the areas the team is weak. Let the new person come in and do the things they like to do and are good at!

[Retric]

Coming up with your own tasks means understanding the problem implicitly. Solving other peoples trivial problems means searching for the interviewer’s hidden gotchas and unspoken requirements.

Ie if someone asks you to find the median value of an array the number of elements and how to handle nulls isn’t obvious, but when you want the median value in an array the context is clear.

[hnfong]

> Normal interviews are mostly a scripted performance where people largely regurgitate answers to predictable questions.

Ah, so you prefer being asked "where do you see yourself in 5 years?", and regurgitating the text generated by ChatGPT instead?

Alternatively, as I replied in the sibling thread, do you prefer to be handed a difficult take home task that is expected to take 2 weeks to complete?

I think I can live with the first, but I'd be insane to work 80 hours on a task just for a small chance to get a job.

[Retric]

“Where do you see yourself in 5 years?” is asking a perfectly reasonable question about what if anything you’re working towards. If you think regurgitating something from ChatGPT is appropriate, I can see why you might prefer more limited interviews.

[crooked-v]

> then objectively the other candidate is the stronger one

...if the job is to complete abstract problems in live coding interviews, sure.

[hnfong]

By your logic, there's no point in doing any interviews at all, because it is nobody's job exactly to sit as a candidate in an interview session.

Is that your point? Because I'm getting tired repeating the point about take home tasks in the sibling comments by now.

[kuhzaam]

What are the "all things equal" here, though? I would tend to disagree that someone who can complete the coding challenge faster is the objectively better candidate. Especially if we are talking about leet coding, where there are people who study their asses off and memorize the common leet code problems. Or maybe the coding challenge isn't leet code, but just so happens to be something that the candidate has done before, or has done recently. Just because one candidate lucked out and has done this very specific thing, doesn't mean they are better than a candidate who could figure it out in half a day, if given a little more time to do so.

[hnfong]

You people are so bent on arguing the unarguable that the arguments are so unnecessarily argumentative.

The interviewer and the hiring company does not have a God's eye view of the candidate and must work within the limitations of the hiring process. The best one can do (without resorting to metaphysical processes) is to make a decision that has the highest expected value (or highest chance of hiring a good candidate).

The fact that there could be a candidate that lucked out because they were prepared for the specific set of questions you asked is not specific to live coding problems. You could have asked them anything and they could have lucked out and just happened to be prepared for the answer you were expecting.

That said, I can tell you for certain that if given a task that the average candidate takes 10 minutes, and a candidate completes it in half a day, you're looking at a 0.1x developer right there. That's by definition.

[robin_reala]

They’re only objectively the strong candidate if that’s the sole thing the company values. It’s unusual for that to be the case.

[hnfong]

With the "all things equal" qualifier, it's sufficient that completing the task faster is a positive signal rather than a negative one.

I mean, this is very elementary.

[tomjakubowski]

a single example of a one minute difference in solving a problem that takes ~10 minutes is noise

[hnfong]

Yes, so given that all things are equal, are you going to hire the slower person then?

Or are you going to, as I mentioned in sibling comments, give them a take home task that takes two weeks to complete, just to be sure?

[codeflo]

I’m also most often on the hiring side of this. If novel problem solving is what you’re actually testing, I would agree. But too many interviewers just think they’re testing that, when what they’re actually testing is whether a candidate has seen a certain memorizable trick before.

I’m talking about the “detect a cycle in a linked list” kind of question. If you ever actually need to do that in practice (though I would question the choices that lead up to that), it’s easy enough to google. The hard part in practice is figuring out that your messy practical problem decomposes into an algorithmic question, but that skill is rarely tested in interviews.

[kps]

Detecting a cycle in a directed graph is useful (in certain domains I've worked in), but doesn't have a cute trick answer.

[Edit: I in mind ‘Detecting the cycles’ rather than ‘Detecting a cycle’ but wrote the wrong thing. Mea culpa.]

Your actual job will be more like implementing a maximally performant directed graph in safe Rust. (Not really. It won't be that interesting.)

[addaon]

> Detecting a cycle in a directed graph is useful (in certain domains I've worked in), but doesn't have a cute trick answer.

Doesn't the standard cute trick answer for a linked list (tortoise and hare) work fine? Do two parallel breadth-first traversals of the graph starting at the root node, one at twice the speed of the other. (If there's multiple root nodes, introduce a unique super-root that points to the roots.) If there's at least one cyclic path, both iterations will enter the same cyclic path because iteration order is the same, at which point it reduces to the linked list problem.

[kps]

Yes, I phrased my comment incorrectly; I had in mind identifying (all) the cycles, and that for a linked list they're the same thing. For a DAG it doesn't generalize because you can't find more cycles without storage to exclude the earlier ones.

[addaon]

Fair enough. I imagine a simple (but O(cycles) space, not O(1)) way to do this is to still tortoise-and-hare, and on detecting a cycle, break the link behind you (by recording it in a table) and continue the BFS. If O(cycles) = O(n) this isn't terribly interesting compared to other approaches, but if O(cycles) < O(n) it might be helpful.

[addaon]

Actually, now that I spend more than ten seconds thinking about this, I think this requires O(1) space; you only need to remember the single most recent link behind you to avoid following it again. Remembering a single break is sufficient to get you to either the next cycle or (if no more) terminate the BFS; and once you detect the next cycle you can safely remember only /its/ break since BFS will never take you back to the first cycle. Assuming streaming output (e.g. print identification of each cycle as you detect it -- including e.g. printing each node until you see the break point, which will identify each node that is part of the cycle), this is O(n) time and O(1) space, which I believe is optimal.

[Freire_Herval]

if you can mutate the graph and mark a node as traversed that's the easiest way. If you can't then save the addresses to a hash set. Both are extra memory but you don't have to deal with "parallel BFS," which honestly is just over complicated imo.

With "parallel bfs" comparing "layers" of traversal has runtime cost that effects the big oh so I think the above solution is better overall even though it feels cheap.

[addaon]

But the whole point of the "cute solution" is to solve with O(1) memory; if you're willing to allow O(n) then the linked list also admits more sane solutions.

I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.

[Ancapistani]

I'd argue the point of the exercise isn't to implement an O(1) solution, but to verify that the applicant understands how the logic they're writing actually executes, and is capable of minimizing complexity.

Whether they arrive at an O(1) solution doesn't matter a bit to me. Most of the time, I don't even care if they phrase it in big-O notation - or have even heard the term!

Given the choice between someone who completed the exercise with an O(1) solution in half the allotted time and someone who never wrote a single line of valid code, I'll choose the person who is able to articulate the problem and potential solutions best every time.

It's far easier to search the web and find a "cute solution" than it is to learn how to think about and communicate complexity.

[Freire_Herval]

> But the whole point of the "cute solution" is to solve with O(1) memory; if you're willing to allow O(n) then the linked list also admits more sane solutions.

Yeah but the "parallel BFS" solution is actually slower. Overall people sacrifice memory for speed in these algorithm problems.

Additionally the input is already O(N) so if you count the input as memory the addition of O(2N) doesn't shift it. If you don't count it well you're deliberately removing real world actuality and making the problem harder.

>I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.

It's layer based traversal via BFS. You have to compare the layers produced by the tortoise and the hare. Let's see for a binary tree:

  For traversal: 
  hare: O(N)
  tortoise: O(N)
  There are N total nodes so traversal via bfs or dfs will always be N.

BFS traversal happens in layers of breadth. The height of a full binary tree is log2(N + 1) - 1. The length of the last layer is 2^height. So the amount of nodes in the last layer is (N+1)/2

Because at each stage of traversal from the turtle and the hare we have to compare the "layer" of traversal between the hare and the tortoise to see if they overlap there is a roughly O(((N+1)/2 )^ 2) comparison as we cycle through each node in the layer to see if it exists in the other layer being compared. This reduces to O(N^2) processing time.

So total is O(N^3) because you have O(N) traversal time, and O(N^2) comparison time on each traversal step. You can reduce the layer comparison to O(N) if you save one layer to a hashset and then compare that set to the other layer, but that has a memory cost and only reduces the total big O to O(N^2).

[Freire_Herval]

I forgot to mention, BFS requires a queue. So if the graph is size N and with one root node and all the rest of the nodes as children that queue will go up to size N. So your solution doesn't actually get away from the memory issue at all.

[yobbo]

A fair point; the only reason the questions are asked is because they have known answers. And unless the questions are trivial, their answers are essentially little tricks. Some candidates have already seen the answer, others have not.

But, testing if the candidate has seen a problem before may be a proxy for something meaningful.

[8note]

The most fun interview question I've answered is "list as many unique ways as you can to detect a cycle in a linked list"

[Freire_Herval]

I'm also on the hiring side of this. I follow the process the company gives me.

Honestly speaking, 70% of the candidates can do the job, but I fail 90% because they can't pass the coding question. It's a filter with a serious problem. Yeah it's all we got, but that filter rate justifies a big discussion because it's crazy.

[darth_avocado]

> that is to be able to take a novel problem that they haven't seen before

But with platforms like leetcode and more, the candidates are split into those who have had the time to prepare (and see the problem before) and candidates who can’t put in that time. And most modern interview processes do not give you credit for thinking through it. You only get through if you solve it the right way, and fast. There is no time to “think and solve a novel problem you’ve never seen before”.

[JohnFen]

> And most modern interview processes do not give you credit for thinking through it. You only get through if you solve it the right way, and fast.

Interview processes that do it that way are fundamentally broken. I wouldn't want to work at such a company. Interviews are two-way streets, and that would be a case of the company failing the interview.

The real benefit of this sort of interview practice is not learning whether or not the candidate can arrive at the correct answer. If we're at the point where this level of interview is even happening at all, you have (or should have) reasonable confidence that the candidate is capable of solving the problem. Whether or not that they do so in the interview is irrelevant. The real benefit is in being able to see how the candidate thinks through the problem.

While I think these whiteboard exercises are not the best approach, when I've been the interviewer at companies that required it, I chose a very difficult problem, gave a time limit, and told the candidate that I do not expect them to actually arrive at the solution within the given time. I just want to see how they approach the problem.

[adamc]

I used to do this in interviews -- give people problems not because I cared whether they got "the" answer, but because it gives you a chance to see their brain work.

The problem with this is that many, many interviewers do this badly. They think they are looking for someone who gets the right answer. Or they are not good at assessing how other people break down and analyze problems.

It is artificial. All interviews are artificial. This is why everyone would prefer to hire candidates they already know a lot about. But a good interviewer isn't looking for a perfect performance.

[fasterik]

I think there's an analogy here to GRE scores and other standardized tests. With GRE scores you can predict someone's GPA, but you can't necessarily predict their success at doing original research. That's because success as a researcher depends on personality traits that are hard to measure like creativity, independence, self-discipline, perseverance, and passion about a subject.

Programming puzzles are the equivalent of the GRE for programming jobs. It might tell you something about the candidate, but I'm skeptical that it measures the most important traits and skills that really good programmers have. It's an empirical question, but your skill at solving programming puzzles might not predict your ability to systematically break down large problems, write readable, robust, well-documented code, or design and debug complex systems.

I also have my own theory about this which might just be my own personal bias. I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. On the flip side, you are selecting against individuals who are intrinsically motivated to learn and build things, but who may not care about those external factors. My theory is that organizations with too much of the former and too little of the latter are going to have issues with groupthink and will be less likely to be innovative.

[hypefi]

"I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. "

Totally agree

[eesmith]

> be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way. ... A contrived live coding exercise gives you a strong signal on this

The author suggests take-home coding exercises with "maybe ... a live call to discuss their solution" as giving a stronger signal than a live coding exercise.

The author also suggested to "present them with a complex technical problem that requires architecture-level decisions across various parts of a technology stack, and have them talk through how they would approach it" as a better signal than a live coding interviews.

So even if what you say is true, you aren't presenting a devil's advocate position. Instead, it makes it seem like you think the alternative to live coding is no assessment, which the author very clearly argues is not true.

[yieldcrv]

> but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.

Its more that businesses don’t know what type of programming they do and are just hazing people for no reason

More like one person is just hazing people due to a strongly held unobjective opinion like yours

Don’t know the better option, just the limitations of this one

[throwaway675309]

I think we're seeing the consequences of not having a properly regulated field that you might see another more strongly regulated engineering disciplines such as electrical and mechanical.

Since there is no CS equivalent of a professional engineering exam, employers have no guarantees that applicants will meet minimal qualifications, and so we continue to see an arms race of increasingly complicated whiteboard / leetcode interview processes.

[gymbeaux]

Is there any other means of evaluating for this skill?