[sgarland]

One problem with this is that it presupposes that the interviewer is skilled enough to spot problems, ask the right probing questions, etc. Casey is definitely skilled enough. I can’t say that’s universal.

Another problem, which TFA hints at, is bias. System Design interviews are often terrible for this reason. If you present me with some scenario that I know is trivial to handle with a single server, I’m going to want to discuss that, but you are probably expecting me to talk about a message queue, a caching layer, etc. Both are valid depending on the situation, but if you’ve only ever known one type, you may dismiss others out of hand.

[jvanderbot]

If you're presented with a system that can be handled by one server and can discuss that architecture, but think they may want to hear about an AWS word salad, just say there's two approaches, discuss strengths and weaknesses for 2 minutes, and ask what they want to talk about. That is precisely the kind of thing people want from a co-worker.

[weavie]

Yes, a discussion of the tradeoffs of different solutions is exactly what I want to hear in an interview.

[thelittlenag]

I've now done probably close to 100 system design interviews. One of the main things I've looked for in candidates is their ability to identify, communicate, and discuss trade-offs. The next thing on my checklist is their ability to move forward, pick an option, and defend that option. Really nimble candidates will pivot, recognizing when to change approaches because requirements have changed.

The goal here is to see if the candidate understands the domain (generic distributed systems) well enough on their own. For more senior roles I look to make sure they can then communicate that understanding to a team, and then drive consensus around some approach.

[sgarland]

> For more senior roles I look to make sure they can then communicate that understanding to a team, and then drive consensus around some approach.

This is why I’m stuck at Senior lol. I can craft incredibly deep technical documents on why X is the preferred path, but when inevitably someone else counters with soft points like DX, I fall down. No, I don’t care that the optimal solution requires you to read documentation and understand it, instead of using whatever you’re used to. If you wanted to use that, why did you ask me to do a deep-dive into the problem?

[AnimalMuppet]

In the real world, the answer almost always is "it depends".

[hrimfaxi]

This only seems to be in software engineering. When I was told me wanted to evaluate a new task queue service, I asked what our constraints were. I was told to survey all options and present a roundup and ignore constraints. Contrast with something like, I don't know, building a house. Do architects consider all possible material choices for a given location, or do they instead limit their consideration to materials that would be suitable to the given environment?

Making the dependencies of "it depends" explicit is the whole point.

[bluGill]

Having built single family houses before I can tell you that architects consider only the choices they know. That is why were I live there are 1000 stick framed houses for every one built with something like SIP (structural insulated panels), or ICE (insulated concrete block) even though the others are similar cost to build with overall and have some useful advantages for the customer that often would make them a better deal for the homeowner long term.

This also means there are 100 builders in my area who only build stick frame houses and won't even talk to you if you want something else and only 1 or 2 who will even think about those other options. (they do compete with the other builders so costs are not unreasonable)

[the_af]

This tracks with my experience with architects (for home renovations): they use what they know, sometimes stubbornly so and even when I must point out it won't work out (e.g. materials inappropriate for the climate, inappropriate due to exposure to direct sunlight or water, obsession with visuals over practicality, etc).

I've dealt with enough architects by now to know this is the rule and not the exception.

[Eridrus]

Maybe I am just bad at interviewing people, but I have tried giving the experiential interviews Casey describes, but I find it quite hard to get signal out of them.

You run into questions of how well a candidate remembers a project, which may not be perfect. You may end up drilling into a project that is trivial. The candidate may simply parrot things that someone else on the team came up with. And when candidates say things, you really have no way to understand if what they're saying is true, particularly when internal systems are involved.

I have found system design interviews specifically much much better at getting signal. I have picked a real problem we had and start people with a simplified architecture diagram of our actual system and ask them how they would solve it for us. I am explicitly not looking for people to over design it. I do give people the advice at the start of every skills interview tp treat this as a real work problem at my startup not a hypothetical exercise.

I have had a lot more luck identifying the boundaries of people's knowledge/abilities in this setting than when asking people about their projects.

And while everyone interviewing hates this fact, false positives are very expensive and can be particularly painful if the gap is "this person is not a terrible programmer, just more junior than we wanted" because now you have to either fire someone who would be fine in another role if you had the headcount for it or have a misshapen team.

[JohnBooty]

    I have found system design interviews specifically 
    much much better at getting signal. I have picked a 
    real problem we had and start people with a simplified 
    architecture diagram of our actual system

To me, this heavily biases towards engineers that have already built or at least designed a similar system to the one you're presenting them.

I believe this will tend to be true even in the ideal case, which is when the interviewer is focused on "is the candidate asking great questions and modifying their proposed design accordingly" rather than "is the candidate coming up with the 'right' solution."

Because, if the candidate his already built/designed a similar system, they will naturally be able to ask better and more focused questions about the design goals and constraints.

     I have tried giving the experiential interviews Casey 
     describes, but I find it quite hard to get signal out of them.

     [...] when candidates say things, you really have no way to 
     understand if what they're saying is true, particularly when 
     internal systems are involved.

Okay, here's where I think I would tend to disagree in a very specific way. First, I would point to the linked interview where Casey conducts a brief mock drilldown interview with the host.

https://youtu.be/gZ2V5VtwrCw?t=2068

The host discussed work he did for Amazon on an event ticket ordering system. Now, I don't think you need to know much about the ticket sales business to think about the challenges here!

The most obvious challenge to me is some sort of distributed lock to ensure the system doesn't don't sell the same seat to multiple people. Another obvious challenge would be handling very bursty traffic patterns, ie an onrush of people and bots when popular event tickets go on sale. Another challenge, brought up by the host, that I wouldn't have thought of is that ticket sales need to avoid "fragmentation" as much as possible. Most people don't want single seats, so you want to sell tickets in a fashion that doesn't leave scattered, "orphaned" single seats that nobody wants to buy.

Those are interesting challenges, and if the interview is an experienced developer, I don't think a candidate could really bullshit through them to a significant degree.

    The candidate may simply parrot things that someone else on 
    the team came up with.

I personally wouldn't care if the ideas originated with the candidate vs. another member of the team. I'd be looking for how well the candidate understood the problem and the solution they implemented, the other solutions considered, the tradeoffs, etc.

    You may end up drilling into a project that is trivial

This feels trivially avoided. If I was the candidate and the interviewer picked a boring project when I had juicier ones, I would just say so. And if I was the interviewer and a candidate told me that, I'd happily pick one of the juicy ones. The point isn't to rigidly lock into a particular project.

    You run into questions of how well a candidate remembers 
    a project, which may not be perfect

This would be expected. Again, I'd be looking big picture here. Approaches considered, tradeoffs made, the "unknown unknowns" that popped up during implementation and how those were handled, etc.

[Eridrus]

> To me, this heavily biases towards engineers that have already built or at least designed a similar system to the one you're presenting them.

Yes, this is not an IQ test, we are trying to see how people react to problems in our domain not measure some generalized form of reasoning. The advantage of picking a problem as close to our real problems as possible is that I don't have to worry how they generalize from the interview to work.

In general, my experience with system design interviews is that people make bad designs, and when you drill down on them they give bad rationales. Similar to coding screens, people just out themselves as not very good at their jobs regularly.

> Those are interesting challenges, and if the interview is an experienced developer, I don't think a candidate could really bullshit through them to a significant degree.

It's not really about "bullshit" per se, but about whether their understanding of their context is correct or not. They can tell you fully reasonable sounding things that are just wrong. In a mock interview, you can see if they ask good questions about their context.

> I personally wouldn't care if the ideas originated with the candidate vs. another member of the team. I'd be looking for how well the candidate understood the problem and the solution they implemented, the other solutions considered, the tradeoffs, etc.

I totally disagree with this. It is very different to be able to remember the design doc for the project and parrot the things that were talked about vs actually writing it.

If I want to hire someone who can design things well from scratch and I get someone who makes bad decisions unless someone is supervising them, I will be very disappointed.

In general, I have given both interviews to the same candidate and after saying a bunch of reasonable things about their existing work, when I ask them how to do design something I quickly find that they are less impressive than they seem. Again, maybe I'm bad at giving experiential interviews, but being hard to administer is a point against them.

My experience of hiring is also that I am generally not looking to take chances, unwinding bad hires is super painful.

[MomsAVoxell]

>It presupposes that the interviewer is skilled enough to spot problems, ask the right probing questions, etc.

Actually, this is a case of Peters principle[1], clear as day.

Recruiters should not recruit programmers if they, the recruiters, have not worked in a programming context. The best software development recruiter is another programmer.

It also extends, of course, to technical managers and is another factor in how one should approach recruitment interviews - is this placement going to result in a high Peter principle, and if so - is the candidate going to be capable of rising above it, or dealing with it in a way that is conducive to the organizations goals?

Because Peter principle is not always a 'negative' - its more of just a condition that results from a lack of communication between parties who really should know each others jobs, better.

People who know how to understand other peoples jobs as well as their own, work great together.

[1] - not Flynn effect

[gedy]

> Recruiters should not recruit programmers if they, the recruiters, have not worked in a programming context.

Ran into this the other day, a company reached out, and while I wasn't job hunting, the senior role they were trying to fill was basically exactly a perfect fit for me from a skills and background on a Venn diagram.

The first two calls were young, non-technical women (they shared linkedin links), and they clearly did not understand what they were hiring for and couldn't answer questions. They insisted on their scripted questions, and didn't want to talk about the companies where I did the exact role they were hiring for.

I was not rude or arrogant about this but the next day got the "Unfortunately, we've..." email. It's actual pretty funny, I'm just glad I didn't really need the job.

Companies, stop letting HR be your first contacts and screening before technical folks. It doesn't work. No wonder your pipelines are full of the fakers and liars like many of you lament.

[MomsAVoxell]

>The first two calls were young, .. they clearly did not understand what they were hiring for and couldn't answer questions. They insisted on their scripted questions, and didn't want to talk about the companies where I did the exact role they were hiring for.

Indeed, this is what I believe is endemic to the recruitment industry - the scripts are there to get the "# of interviews done" statistic in line with some expectation; the interview itself being successful, is secondary to the purpose.

I have personally saved myself huge headaches by flunking a recruiter who wasn't really doing the job, just pretending to.

[the_af]

I dunno, I find non-technical screening helpful. Otherwise way too many candidates reach the tech interview stage that shouldn't, and are a waste of my time...

(I agree screening by non-programmers is flawed, but there's always a compromise. Involving programmers at this early stage can be a source of frustration and wasted time).

[gedy]

My point would be that engineering manager, etc are the ones who vett folks to be talked to. I'm pretty sure every job I've gotten in 25 years was initiated by engineering lead or manager.

[MomsAVoxell]

The landscape is large and wide - various realms of software dev/engineering are walled gardens compared to others, which are more public markets.

Imagine the recruitment industry around Munich or Melbourne or Singapore. Similar fish, very different kettles.

[the_af]

Hm, interesting.

In my career, some have and some haven't. In my current job, there's a screening by non-tech recruiters, then managers, then me and possibly others for the real tech interview. My EM is technical but cannot go in-depth as we can.

[kube-system]

Recruiters are fine at recruiting but they shouldn't be a substitute for a technical evaluator.

[the_af]

In my experience, recruiters seldom recruit programmers, they just find them and do a screening, but there's a technical interview done by technical people.

Are there companies that skip the tech interview step?

[xboxnolifes]

The screening is part of the recruiting.

[the_af]

Yes, but the comment I was replying to and disagreeing with was this:

> Recruiters should not recruit programmers if they, the recruiters, have not worked in a programming context. The best software development recruiter is another programmer.

I don't think recruiters must be programmers, or even that that's a good idea. Recruiting and doing the tech interview are different steps.

[MomsAVoxell]

A recruiter who doesn't know what a good programmer does and how they work, won't necessarily recruit good programmers - but indeed, rather leave that identification to the tech interview - which is why a tech-savvy recruiter can offer a better service than a non-tech-savvy recruiter would, by removing a seat in the game of building a great development team/groups/department, at scale.

Also, this is changing now, radically, with AI/ML tooling - we programmers can indeed cut out a lot of middlemen now.

"recruiters must be programmers" - this absolute is not a policy, rather a test against the nature of the relationship being offered. If I, as a developer, am mis-represented by the recruiter to the tech interviewer, then it is an inadequate recruitment.

If however, a tech-savvy recruiter does the tech interview and understands the nature of the systems being built/technology needed for specific cases, then the process is smoother - not just for the company, but also for the individual.

This pre-supposes, of course, that the interview candidate is taking the process seriously, and identifies faulty onboarding practices. If you're not asking pointy questions about why a recruiter is being involved, or finding out whether or not the recruiter understands your Resume/CV well enough to represent you, then you're putting yourself on the wrong foot.

Developers/technologists being managed by non-devs, is a huge source of friction - and bloat - in the software industry.

[jonathrg]

What is Flynn factor?

[MomsAVoxell]

I meant Peter principle:

https://en.wikipedia.org/wiki/Peter_principle

[IAmBroom]

So you meant to talk about "Peters principle", not "Peter principle"... I can't find anything on the first, that isn't really the second.

[MomsAVoxell]

It really is applicable, whether you say "Peter/Peters".

The point is that if you're getting interviewed by someone who cannot understand your job, they are in the throes of a Peter principle manifestation, and you are very likely to end up in the same situation if you don't identify the incompetence - you'll inherit it.

[msluyter]

A third problem is that, depending on the project, one's recollection might be pretty fuzzy. A fourth is that, while you might be a great programmer, perhaps you've never had the opportunity to do the work of the design or greenfield development, meaning, you may not have a ton of insight into the design work that went into the project. E.g., perhaps you mostly do maintenance work on a large number of projects, so your overall knowledge of each is fairly shallow.

[pton_xd]

Recollection of specific implementation details of an old project may be fuzzy, but I'd hazard that any competent programmer should be able to discuss the themes and challenges of the project in depth, along with approaches for different requirements.

Overall shallow knowledge is not a positive signal, in my opinion. If they really are a firefighter who constantly jumps around, the interviewer should lean in to the organizational challenges they face when identifying and fixing problems across a variety of projects and domains. There's always a way to drill down with more specific questions.

[JohnBooty]

Yeah. Even a "firefighter" should be able to discuss the architecture(s) they inherited, and the associated strengths and pain points, and the tricky/interesting parts of the system. So I wouldn't view that as an impediment at all to this style of interview.

And I'm not interested in the sorts of specific details that go fuzzy over time. I'm interested in a big picture view of choices made and alternatives discarded. We should talk about some implementation details as well, but "I don't recall" is definitely an acceptable answer for some decently large percentage of those choices.

If you're talking about a Rails project you did ten years ago, I don't really care if you used Sidekiq instead of Resque or vice-versa. I mostly want to know if you know what sorts of jobs should be moved to background tasks and how you structure those jobs and what are the tradeoffs etc.

Also, if I (the interviewer) selected one of the candidate's more "boring" roles, I'd happily let them suggest we focus on one that is juicier and/or one where they were more involved in the design process.

[pelagicAustral]

I think this model fits a lot better to mid and small size businesses. FAANG et al will always need their leetcode barrier due to the sheer amount of interviews they have to conduct. I wouldnt think you can spare an engineer that can go through this type of interview on a daily basis, at that point the person will be solely conducting interviews and not coding at all.

[bluGill]

FAANG et al should focus on employee retention a little more so that they don't have to interview as many. If they keep people around for years not only do they have less interviews to do, but they also have incentive to care about quality. There should be no gain to individuals moving jobs every few years. There should be gain to the company keeping experienced people around.

[callamdelaney]

I agree with your latter point on system design. I've had 'big data' interviews where the 'big data' fits in the memory of my desktop machine - so I'm going to give you a very different answer to your over-architected hadoop cluster running over 20 1gb containers or something.

[jvanderbot]

You have to make this a back and forth, not a monologue. Nobody is going to criticize you for tailoring your response to what they want to talk about. Treat it more like a coworker you respect asking you about architecture options, not like a right/wrong graded quiz from college or a chance to assert your knowledge and experience. Your knowledge and experience will show anyway.

[__alexs]

I agree but it is not uncommon to find interview processes where a monologue is exactly what they expect and they are unable to have an actual conversation about the topic for whatever reason.

[jvanderbot]

Then it would especially behoove you to spend 2 minutes to figure out which monologue they want.

[dangus]

Maybe this is a dumb disqualifying example, but if you’re hiring someone who has worked for the government for the last 20 years on classified projects, this whole method has to be thrown out the window.

[CoastalCoder]

I think it depends. A careful reading of the classification guides might let a candidate discuss a lot of technical details.

OTOH, I'd hate to be in a position where I had to think carefully about every aspect of that stuff during an interview. Even if nobody noticed an inadvertent classified spill, it could still suck.

[SamInTheShell]

No. You may not even talk around the topics. This explicitly comes with the training. Anyone asks me about my last job, I don’t really state much more than what the job title was and what the job postings said for the role I applied to.

[CoastalCoder]

I suspect your training went beyond what the law requires, but perhaps I'm mistaken.

[bluGill]

I work with people who have worked on various classified projects in years past. Once in a while someone will mention something about the project they read in a newspaper and they have to say "It is obviously unclassified now but because I wasn't explicitly told that is was no longer classified I am not allowed to talk about it" - then they change the subject.

[kridsdale3]

This applies equally to every employee at Apple.

[ModernMech]

Yeah, I was once at an Apple interview and they couldn't tell me what I'd be working on if they hired me (I assume it was a mobile robot, as my background is in robotics motion planning). I politely left the interview after the third one tried to keep me in the dark and insist whatever it was it was world changing and bigger than iPhone. To me, job interviews are a two-way street, but they wanted to assess me while not allowing me to assess them, so I felt like they had wasted my time.

[dangus]

In some fairness to them, getting your specific job is less important than avoiding a lawsuit from Apple’s rather aggressive legal department.

I definitely empathize with your perspective. Simultaneously, I can also see how it would be most ideally fair to candidates to have some method of evaluating them without relying on intimate details of recent job experience.

E.g., let’s pretend that Steve Wozniak took 5 years off to raise children, are you going to pass him up because he has no job experience from the last 5 years? You’d be well within your rights to, but as a hiring manager I want to choose the best candidate, not the candidate with the best resume.

[alyxya]

IMO the biggest limitation is that there's a lack of feedback in being a good interviewer. It's hard to get better at something you can't measure how well you did in any meaningful way.

[Eridrus]

At my company, we do interviews remotely and record them and then sometimes review them and give feedback.

[ch4s3]

I once failed an interview because they asked how I’d solve a geospatial problem in Postgres and I said I’d use PostGIS and order by ST_Distance. The guy said “what’s postgis” and I knew it was over.

[WesolyKubeczek]

You failed, or was it them?

[ch4s3]

I believe we all failed as an industry that day. A friend of mine worked at the company and tried to get me to reapply a year later and promised they had scrapped the old interview process, but I just couldn’t.

[renewiltord]

Wait. What’s the “right” answer? The “in Postgres” naturally directs towards PostGIS.

[ch4s3]

They apparently wanted me to do some geometry, and know that the naive geometry doesn't account for the curve of the earth, and that there are indexing concerns. This was for a company that makes HR software and the interviewer who devised this exercise had never heard of PostGIS and sort of implied it was a real thing that people used.

[renewiltord]

Why on earth say the Postgres part then? Bullet dodged.

Everyone in GIS knows all the geometry stuff, but also if you say "in Postgres" you're strongly implying PostGIS.

[ch4s3]

And for an HR web app that doesn't do any geospatial stuff...

[kube-system]

An unskilled interviewer won't do a good job using any technique.

[rvz]

> One problem with this is that it presupposes that the interviewer is skilled enough to spot problems, ask the right probing questions, etc. Casey is definitely skilled enough. I can’t say that’s universal.

That is the entire point of the technical interview.

Someone who is already experienced in a particular subject is also at least qualified to ask the right questions to the candidate related to the subject.

[epolanski]

The another problem is the same #1 problem.

[faangguyindia]

why even interviews are necessary?

It all can be solved by a system where 1 company hires programmers then other companies can rent specific programer by hour and leave reviews.

This way a programmer doesn't need to interview, he's always employed and getting paid.

If they find no company to bid on your time, they can fire you after sometime.

[nwalters512]

A lot of (most?) useful work requires the kind of domain-specific knowledge that can only be built up over a lot of time spent working on a given team/product/codebase. "Renting" programmers here and there on an hourly basis might work fine for basic tasks, but would probably end in frustration for both the employer, who has to deal with a constant stream of random new people, and programmers, who are constantly bouncing from project to project.

[bluGill]

That depends. There are a lot of things a "by the hour" programmer can do when guided by domain experts. However a domain expert can only guide a couple such people and you lose their ability to get anything else done.

[walkabout]

The largest tech companies could also implement a shared cert program via some kind of industry group, with periodic re-certification requirements. This’d surely be far cheaper than all of them running multi-stage leetcode and system design interviews, and they already basically publish study guides so this isn’t even that different, just cheaper for both the businesses and candidates.

It’d also make job hopping far less painful (see above: cheaper for the candidates) which is why they don’t.

So we may conclude: the point of leetcode interviews is wage suppression.

[ModernMech]

Never gonna happen, as they outsource this certification to universities already. They're not going to stand up their own cert programs to streamline the process; the muli-stage leetcode interviews are about hazing culture and making sure employees are servile, not merit culture and making sure employees are competent and qualified.

Like you said, tech companies need candidates to feel like they barely passed a grueling interview because it makes them wary to jump ship and have to go through that again, not that they are well qualified industry professionals who have the credentials to move between jobs and work anywhere they are certified.

[kube-system]

You have reinvented software consulting. There are many criticism to that model as well.

[ttoinou]

   If they find no company to bid on your time, they can fire you after sometime.

In most of the world you can’t fire people like that

[hansvm]

> rent programmers by the hour

That's much less efficient for big software projects. The knowledge built up over time is one of the core things making you effective, so you have a big productivity loss if you don't retain the same people for a long time. With a continual middleman, the overhead from that "hiring" process is usually proportional to work done, so after a period of time it will have been more efficient for the employer to hire directly (even at a cost of tens of thousands of dollars) rather than to go through the middleman.

> leaves reviews

We've seen how well that works in every other part of the market.... Interviews are necessary because of a lack of trust. In your system, every party has an incentive to lie, and the incentives are strong enough that it's worth paying people to facilitate your lies. You see that with various contractors all the time, having somebody with good reviews take the contract and somebody with fewer skills actually do the work, the employer understating the work to be done, etc. Good people exist, but they're hard to find in a low-trust environment.

> always employed and getting paid

Interestingly, that usually benefits everyone except the programmer. As a rule of thumb, the person taking risks will have higher returns, so if you can afford those risks you should choose to take them yourself. Examples include various forms of device "warranties" (if you're paying, it's insurance, not a warranty, and I can afford the financial hit when a thumb drive fails, so I'm not going to pay a premium to insure against that possibility), some forms of actual insurance (full coverage on used cars is an example -- if my car is ever totaled I'll just buy another -- I have liability insurance out the wazoo, but full coverage isn't worth it), etc. Programmers often make gobs of money, so except for potentially the very beginning of your career you should be able to easily weather a few years without work. The exact contract details vary, but that would suggest then that programmers are better off on average cutting out the middleman providing those risk guarantees (and they're not really guarantees in the first place, right? people are currently without jobs because there are fewer programming jobs than there used to be, and no middleman scheme is going to fix that; you have to wait for this economic blip to blow over).

I could go on. Contracting is fine, but it's not a replacement for all salaried work.

[IAmBroom]

We have that already. It's called "temp hires". Mostly it just screws the employees, while the middle man gets richer.

I can't think of a reason why a highly skilled individual would sign up for that.

[creer]

Either somebody somewhere screens for competence, education, skill, talent, even bare willingness to do the work - or the project will suffer. Adding a level of indirection doesn't wolve that.