[PaulHoule]

I like the bit about

   Let's look at how the first one of these, “pay attention to weak signals”,
   interacts with a single example, the “WTF WTF WTF” a new person gives off 
   when the join the company.

and kinda wonder if a company that prioritized not getting this reaction from new hires might find it is the most impactful thing they can do in terms of culture.

[jkaptur]

Just to give a different, concrete, perspective (and push a hot button HN issue), I've spent a fair amount of time working on extremely large web applications, and by far the #1 "WTF WTF WTF" thing that new hires say is "what do you mean you aren't using $TODAYS_HOT_JS_FRAMEWORK??"

Once you get away from "should we use version control" and into actually difficult software engineering questions, it's not clear how to balance a fresh perspective vs. an experienced (normalized? tainted?) view. I wish the article went into this more.

Like, how does the new hire (or anyone else) know the difference between "learning the complexity of the new system" and "internalizing/normalizing the deviance of this culture"?

[aidenn0]

> Like, how does the new hire (or anyone else) know the difference between "learning the complexity of the new system" and "internalizing/normalizing the deviance of this culture"?

If a new hire can't checkout, build, and test the software on the first day, then there is likely something either wrong with the hire or the infrastructure. A sufficiently old and arcane software system might take weeks before a new hire can make even a simple change, but that shouldn't impact those three items.

[Jtsummers]

Along with this "how long to spin up the new hire" issue, one of my first (if not the first) questions when trying to help people improve their processes related to software is:

> If the user/client asks you to make a small but not trivial change, how long would it take to update and deploy the program?

I have had answers ranging from "A couple hours" to "A year" (yes, they were serious). Most were in the 1-3 month range, though, which is pretty bad for a small change. It also makes it apparent why a bunch of changes get batched together whether reasonable or not. If a single small change, single large change, or collection of variable sized changes all take a few months to happen, might as well batch them all up. It becomes the norm for the team. "Of course it takes 3 months to change the order of items in a menu. Why would it ever be faster than that?"

[MikePlacid]

Not sure how speed of the change is related to batches. The “batch” is related to “due this week”, there never was a single item in this list. “Speed” is related to “due which week”, that depends mostly on the priority of the change, not on how easy it is.

Upd. And “change menu items order, fast” is a sign of a problem. We found Mac Cube in ski vacation rental home once. It ran MacOS 10.2 or something. All the menu items were in the places we expected them to be! You think carefully first, then you implement menu items order. Upper Apple -> About this Mac. We managed to break their network config in like 5 minutes!

[Jtsummers]

By running the exercise with a small change the constraints and behavior of the rest of the process get emphasized. As an example, in one team their test process was entirely manual and took a month. They ran that entire test suite for every release, whether the release should have affected the requirements being tested or not. Why? Mostly because they didn't know what changes in the release would affect what requirements, but that was another problem. This did encourage larger batch sizes though because if you have that large cost in your process and the cost is fixed regardless of batch size (100 changes or 1, you spend a month testing) you might as well batch more changes into the release. Having more releases means you incur this large fixed cost more often and overall reduces your throughput.

And I don't think I understand your update to your comment or you don't understand the point of that example from mine. It was illustrating the submission topic: normalization of deviance. Sure, you should think about where things should be but if a customer comes in and says, "Swap these two items" and you can't provide a working version with that single change for months then things have gone off the rails somewhere. I put it in quotes to reflect a statement like what I have heard from those teams I worked with. To them a long effort for a trivial change is normal, when it should be considered deviance.

EDIT: effect->affect. Always trips me up.

[MikePlacid]

> As an example, in one team their test process was entirely manual and took a month.

Ouch!

[aidenn0]

This is true, but there are cases where "months" makes sense. E.g. avionics, industrial, medical.

[Jtsummers]

That can extend the time, yes. I had a caveat about that but apparently edited it out before submitting. But that only explains or justifies the delays if there is value added. I work in aerospace and these were avionics and related systems. The bad ones did not have quality as a reason, though the good ones did. The bad processes were swamped with manual testing (which was non-comprehensive and error prone) or really tedious CM related activities (which was manual and error prone).

You can do a lot with good test automation, even in avionics. That cuts down a ton of the time and usually improves quality.

I'll also note, don't take my "deployed" too literally. I used that term because so many people here are working on server-based applications where that makes sense. Think "out the door". The exercise can only go as far as the team/org's reach. Deployed for avionics would mean more like, "At the flight test team". After that, it's up to someone else to schedule it and get it returned with issues or fielded.

Going beyond the team's reach without including those people (and thus making them part of the team, after a fashion) is guess work and opens up the blame-game. "It's all flight test's fault it takes a year to get out to the customer." Well, it takes you 9 months to get it to flight test and them 3 months to get done. So why does it take you 9 months? If you have a good reason (complex system, lots to test) then that's valid. If it's a simpler system, 9 months to get it to flight test is probably not justifiable.

[aidenn0]

I totally agree that "process" on its own doesn't justify long turnarounds. If you had one part of your code that was taking 2/3 of the run-time, it should get plenty of scrutiny, and the same is true of processes.

[charrondev]

For us the setup is:

- Install docker - Setup GitHub SSH credentials. - Pull the main repo. - Run a script that will pull down related repos, install dependencies, start up a bunch of docker containers, and then run health checks on the app. - setting interactive debugging takes a bit longer, but not too much more.

Unfortunately I’ve routinely dealt with our IT department being slow to give credentials to new employees or shipping them under provisioned or just incompatible systems. No you can’t give our new senior developer the same cheap crap laptop running an ancient version of windows on that you send to the junior marketing person doing cold calls all day.

[908B64B197]

> by far the #1 "WTF WTF WTF" thing that new hires say is "what do you mean you aren't using $TODAYS_HOT_JS_FRAMEWORK??"

To that speaks of the caliber of programmers hired. If all they have seen is $TODAYS_HOT_JS_FRAMEWORK and wrote nothing but a web app using $TODAYS_HOT_JS_FRAMEWORK they might not grasp the fundamentals that would make then realize that frameworks are just abstractions (and not that different from one another).

I don't think any software engineer would even ask that question, since the answer will almost always be "$TODAYS_HOT_JS_FRAMEWORK didn't exist when the project started, and it's not worth a re-write to port it over".

Now, that brings out a second important truth: a company can't attract and retain a wide range of different caliber employees. For instance, if a place still questions the usefulness of source control (perhaps because they consider git to be too complicated) there's no way they'll attract and retain top performers. So the culture will select people that agree that source control is a waste of time.

[blep_]

I do not understand how teams work without source control. I don't mean that (only) in the "WTF are you doing, I don't understand why you would do that" colloquial sense. I literally cannot conceive how they work.

Do people just... change files and then email the whole file to the other developers and hope nobody else was working on that file? Do they at least have patches?

[colomon]

First professional software development I did, we didn't even have hard drives or networking. Luckily I was the only developer and it wasn't a huge problem.

First place I worked with other people, we at least had hard drives. I don't think we had networking on the machines or version control. For sure there was only one or two machines in the office that could reach the Internet. Mostly only one person could work on a file at a time.

When we did get more employees, a LAN, and version control a few years on, the mid-1990s Microsoft version control software was such a piece of junk it mostly amounted to a formal digital system specifying who the one person who could work on a given file was...

[908B64B197]

I've seen people exchanging thumb drives.

No concept of a patch. They spent most of the afternoon and evening "performing the manual merge and stabilizing the release", meaning rebuilding and deleting lines until it compiled.

I wish I was kidding.

[Jtsummers]

Email would actually be better than what I have actually seen because it gives a pseudo-version control system, patches or no:

Shared drives and folders with concurrent edits. Sometimes they'd separate them into "mine/2023.02.14" and "yours/2022.12.10" but that wasn't much better. Actually, because people don't seem to grok lexicographical order, or how to write dates at all, the dates are normally 10-12-2022 and 2-14-2023, guess whether the first one was from October or December.

[BlueTemplar]

There are less complicated VCSes than git, so this sounds like a cop-out.

[josephg]

> it's not clear how to balance a fresh perspective vs. an experienced (normalized? tainted?) view.

Having a healthy, balanced view comes from enough experiences. Ideally from working in a bunch of places and seeing enough things go sideways, and correctly understanding and identifying the causal chain that led to failures.

Funnily enough its sort of like training an AI - you essentially need a lot of correctly labelled data to learn. Junior engineers don't have enough data points, and unfortunately some "senior engineers" I've worked with took (in my opinion) the wrong lessons from their experiences. (Eg the CTO who thinks version control is too complex.)

The interesting cases are when smart, experienced people disagree on what the best solution is. Should you keep your team small and smart or have a varied team with more mentorship and process? Is code review worth it in every case? What is the right amount of tests for your software? How often do we want to push to production?

When I was teaching programming my students would sometimes ask juicy questions. My favorites were the questions I could answer with "I'll tell you my opinion, but I've worked with people I look up to who think I'm wrong about this..."

[flappyeagle]

If "what do you mean you aren't using $TODAYS_HOT_JS_FRAMEWORK" is the first question they ask, then you can end their employment right then and there. Hire people whose "wtf" you take seriously.

[evilduck]

Both parties would feel like they're dodging a bullet.

People still griping about $TODAYS_HOT_JS_FRAMEWORK are clearly out of touch in 2023. It was funny commentary in 2013. It still rang a little true until around 2016. Now it's just an indicator you are the one not to be taken seriously.

It's React, Vue, or Angular and it's been that way for many years now.

[notduncansmith]

React underlies a lot of the “new hotness” like Next.js and Remix.run (I’m not sure if Vue, Angular, or Svelte have equivalents but I wouldn’t be surprised).

[Cthulhu_]

I still take those newer frameworks with a big grain of salt, but tbh, that's because of ignorance on the one hand, and fear of sunk cost on the other (e.g. spending time to invest in acquiring knowledge only for it to go obsolete).

Angular is still used in a surprising number of large companies my employer works for, so that, and React, and Vue are all solid investments IMO.

The other ones I think will only be used if one developer dares to take and sell the risk.

[whats_a_quasar]

There's a whole section about how to balance a fresh perspective, in "solutions". The best way is for an effective VP to hear the WTFs of new hires, apply engineering judgement, and make changes based on that signal. If management is not the ones creating the deviance, they ought to be able to tell what reactions are just unfamiliarity with the system and what are signs of something actually being broken. The article is arguing that most people ignore those weak signals by default, and ought to pay more attention to them, not that they're always reliable.

[jt2190]

Yes. From the article:

> “The thing that's really insidious here is that [once a person buys] into the WTF idea… they can spread it elsewhere for the duration of their career… Once people get convinced that some deviation is normal, they often get really invested in the idea.”

> [H]ow does the new hire (or anyone else) know the difference between "learning the complexity of the new system" and "internalizing/normalizing the deviance of this culture"?

The article implies that the new hire should pay close attention to the things that are incentivized, and those that are not.

[ReflectedImage]

That's not a "WTF". All front end developers trash their predecessors work and rewrite in the last framework. That just what they do.

[GartzenDeHaes]

In my experience, you cannot change an organization's culture with rules, mission statements, listing values, or giving speeches. They only way is to take down the old culture bearers. Some of them may be managers, but more often they are employees who have gained some organizational power. You'll often find them at the center of sticky organizational spider webs with approval processes such as purchasing and service administrators.

To change the culture, these people have to go. Firing them may not be feasible, but there are other options. Dethroning them in the form of a promotion or even just physically moving them can be effective. When people don't have to jump through their hoops anymore, they lose their organizational power.

[Ensorceled]

> They only way is to take down the old culture bearers. ... To change the culture, these people have to go.

I was briefly head of engineering at a company that had several "old culture bearers" that made change impossible. I was something like the 3rd or 4th engineering leader over the space of a year. Apparently the person after me was actually allowed to fire a few of these people and was able to turn things around.

[PaulHoule]

More than once I had to leave in order for an employer to take action against a problem employee, including my boss once. In these cases the employer suddenly realized that it made no sense to attempt to replace me without removing the reason that drove me out.

[Ensorceled]

That is, unfortunately, very common. Though sometimes the toxic people drive the company into the ground first.

[chaps]

Been told that before. When I spoke up, I was told that I was new and shouldn't talk about things I know nothing about.

[giobox]

There's often a wise tradeoff between criticizing systems you've just seen after being at the company 5 minutes and actually spending some time at the company to learn the historical context of why the thing you think is insane/shit is insane/shit before telling everyone who built it how insane/shit it is.

People generally don't wake up in the morning and go into work motivated to make insane/shit things - context, tech debt and business realities all mount up and even the best of us can end up making choices that in isolation look crazy.

There are of course companies who are really bad and you may well be right, but so many times I have seen in my career a young new hire storm in and think everything is shit without paying heed to the context and historical pressures. The best thing you can do in many cases is spend the first ~six months at a new tech company trying to understand that context, and indeed I think more mature engineers generally do.

[Buttons840]

> There's often a wise tradeoff between criticizing systems you've just seen after being at the company 5 minutes and waiting 6 months to understand the context.

I know this is reasonable advice, but it makes me deeply cynical. After 6 months I will have learned to live in the shit (to use your term), and so it still seems like I have nothing to gain by speaking up or trying to fix things. A culture that accepts shitty code probably isn't supper demanding for an experienced developer who is accustom to the mess, so I'll just coast through my time and hop jobs after a few years.

If nobody wants to respectfully talk about my criticisms on day one, then they wont really want to at 6 months either. In the end I'm lead to believe I should have zero concern for code quality and only worry about my personal reputation.

[AA-BA-94-2A-56]

This is exactly where I am at right now. I have a million dollar mortgage and interest rates are going up. I'm keeping my head down and perpetuating technical debt until I can hop jobs for a higher salary.

Criticising the status quo is not a winning move for me, especially when it's lead to the company's engineering team tripling in size. If I'm asked, I'll pick some low hanging fruit– remove reliance on legacy/redundant JavaScript libraries such as jQuery, and spent time writing better unit tests. But so far I haven't been asked.

[chaps]

You (in fashion of the article) missed the point I made: I was explicitly asked to speak up as the new employee and when I did I was told to stop speaking up. When I brought it up in the meeting I gave my two week notice, he admitted to saying that and apologized.

[dec0dedab0de]

That's some bullshit. you did the right thing. When a new employee joins and notices that something sucks the answer should be something like these:

We know, but haven't had time to fix it, maybe we'll assign that to you when you're caught up.

We didn't think of it that way, good catch, lets go into detail later.

Yeah, but doing it this way makes this other thing easier, we'll show you that when you're ready.

or even: I don't know, my brain is fried with this project, can you ask again in a few months?

[whats_a_quasar]

I get what you're saying, but this is exactly how deviances are normalized. When you've been with the company for a long time and are familiar with the history, it's easy to rationalize why things are the way they are and that they can't be improved. You can explain something that's crazy with context and historical pressure.

Dan's point is that sometimes the new person's judgement is correct, and there actually is a real problem that's invisible to people who have been with the project a long time. But the new person's judgment is basically always ignored, and that's a mistake - it ought to be weighted heavily because they legitimately have a perspective that insiders no longer have.

If instead you spend six months trying to understand the context:

"new person joins

new person: WTF WTF WTF WTF WTF

old hands: yeah we know we're concerned about it

new person: WTF WTF wTF wtf wtf w...

new person gets used to it

new person #2 joins

new person #2: WTF WTF WTF WTF

new person: yeah we know. we're concerned about it."

I'm sympathetic because my first company was a mid-stage startup with huge structural problems in the engineering org structure and processes. When I joined I had frequent "WTF" moments and had a similar experience where experienced people would explain to me why things are the way they are. So I trusted them, and put my head down, but eventually got frustrated and left. A few months later the company went bankrupt because they couldn't build product fast enough, investors lost patience, and they couldn't raise another round.

[Buttons840]

> the new person's judgment is basically always ignored, and that's a mistake

Remember, the new person has something that nobody else on the team can ever learn, no matter how much they study or how long they work. The new person has a fresh perspective.

[a4isms]

"Making good decisions, therefore, requires understanding past decisions. Without knowing how things came to be, it’s easy to make things worse."

—https://thoughtbot.com/blog/chestertons-fence

[chaps]

I was once told this after raising the issue that "hey maybe an API that responds with root ssh passwords is a bad idea, and our clients are going to be pissed once they find out." And.. I was right.

So often, citing Chesterton's fence is significantly more naive than what it attempts to criticize.

[a4isms]

How is being right about wanting to make a change a refutation of the idea that knowing why people made a decision in the past is an excellent idea? Nothing about Chesterton's Fence asserts that people in the past always made good decisions, nor does it assert that perhaps what was a good decision then is a bad decision today.

It simply asserts that understanding why a thing is the way it is is valuable when making a decision to change it.

That understanding could be as simple as--to take a real world example that most readers here will remember--"They chose to install a hidden web server on the user's system, because they felt it was the best way to deliver user convenience given the resources and time the team had available."

We can still say it was a bone-headed choice to do that because it opened a massive back door to every user's system. And? What is the problem with looking into why they made that choice before arguing that the choice should be reversed with maximum prejudice?

Chesterton's Fence isn't a suggestion that no changes should be proposed, or that if you look into the original motivations you will change your proposal. Think of it as insurance against the possibility that every once in a while, you will discover a requirement that needs to be addressed with your suggested change.

I don't see where you're coming from that quoting Chesterton's Fence is even "criticism." It's a suggestion to take out a little insurance by doing a little homework.

[lmm]

> Nothing about Chesterton's Fence asserts that people in the past always made good decisions, nor does it assert that perhaps what was a good decision then is a bad decision today.

> It simply asserts that understanding why a thing is the way it is is valuable when making a decision to change it.

The second assertion is implicitly an assertion that decisions made in the past are, if not always good, at least good enough often enough to be worth understanding. In my experience that's not true; most of the time it's just something someone did without really thinking about it.

[chaps]

  We can still say it was a bone-headed choice to do that because it opened a massive back door to every user's system. And? What is the problem with looking into why they made that choice before arguing that the choice should be reversed with maximum prejudice?

Because I was suggesting a different method for the specific task at-hand.

  I don't see where you're coming from that quoting Chesterton's Fence is even "criticism." It's a suggestion to take out a little insurance by doing a little homework. 

Every time I've heard someone quote Chesterton's Fence, it's always been as a means to halt the conversation. Essentially, "shutup" -- an indirect critique of critique itself in dismissive form. There's possibly some meta point here about you not knowing the full circumstances of the situation to warrant bringing up Chesterton's fence.

[justin_oaks]

I agree. Chesterton's fence doesn't mean that if you don't know why the fence is there, don't ever move it, under any conditions. It means try to find out why it's there before moving it.

In many cases in my career, I've seen code that doesn't make sense or seems like a bad idea. The person who could explain why it's there has long left the company. Am I afraid and leave the screwy stuff there, while citing Chesteron's fence? Hell no. I'll change it to do the right thing. This results in either exposing the reason why it's there, or showing that it really was unnecessary/bad. If something breaks from the change then it's good that I can finally document what wasn't documented before. So either way it's a win.

[a4isms]

I readily accept that this strategy has worked for you. In my particular case, I work with customers and software where making a change, pushing it to production, and finding out that I missed a key requirement when it breaks is sometimes unacceptably consequential.

But that may not be true for everyone. If making changes and seeing what does or doesn't break is a successful strategy for you, go for it.

[aidenn0]

I can think of a dozen reasons why such an API might exist. Chesterton's fence doesn't say "don't make changes" it says "If you can't think of why something might exist, you are not qualified to say that it shouldn't exist."

[chaps]

I never said I didn't understand why the API existed.

[Ma8ee]

The problem is that the historical context of a decision often becomes a defence of the status quo, even when most people understand it is bad.

[PaulHoule]

It's not (1) "reacting to the reaction" which is the endpoint but (2) not having that reaction. If (1) is important it as because that is the path to (2).

I'd say a company that has accomplished (2) has cut the workload in hiring employees by 30-50% in the sense that every employee who has reaction (1) either internally or externally is at risk for being disengaged or leaving soon. Not only that but you are probably wasting your dev's time and could get dramatically more productivity out of them if you aren't WTFing them to death.

[AnIdiotOnTheNet]

To be fair, you really shouldn't. You know nothing of the constraints that people are operating under, or the political or cultural landscape you're dealing with, so you just come off like a preachy academic.

[chaps]

See the other comment I made: I was explicitly asked to speak up.

[AnIdiotOnTheNet]

If that's the case, then yeah fair enough. If someone doesn't want your opinion they shouldn't ask for it.

[lo_zamoyski]

That doesn't seem like a healthy standard b/c it grounds decision making in appearances rather than principles and prudential judgements. Certainly, such feedback or opinions can be worth considering as a way of getting at what principles are being violated and deciding whether these violations are tolerable or what ought to be done about them. A fresh pair of eyes could help. But an untrained pair of eyes might also not be qualified to discern the right course of action.

[PaulHoule]

Yes and no. But 2/3 of the time the problem is that the company has no documented build process or something obvious like that . They have time to spend 2 years failing to deliver a product because they don't know how to build it, but when somebody asks "How do we build it?" the answer is "Don't waste our time asking stupid questions." Of course they have been wasting time not knowing how to build the system, the guy who started the project might be able to hit F5 in 15 different windows and get it to sorta kinda work, but new hires they are hiring to work on it are quitting right away and somehow they can never get it into production.

There should be no controversy at all that complete instructions for installing everything required for a dev to build the project and work on it should exist and it should be possible to complete this task in hours, not the weeks that it frequently takes. And, no, "docker" is not an answer to this anymore than "The F5 Key is a Build Process"

https://blog.codinghorror.com/the-f5-key-is-not-a-build-proc...

It is not "Docker" that solves the problem, it is the discipline of scripting the image build process into a dockerfile. If you know how to write a dockerfile you can write a bash script that runs in 20 seconds as opposed to having Docker spend 20 minutes downloading images and then crash because of a typo.

You are right that a company might have good reasons for doing things in an unobvious way, but most of the time when nobody at a company claims to understand what the company is doing except for the CEO and people aren't too sure about the CEO, it is the fault of the company lacking alignment, not a natural property of freshers.

[skissane]

> It is not "Docker" that solves the problem, it is the discipline of scripting the image build process into a dockerfile. If you know how to write a dockerfile you can write a bash script that runs in 20 seconds as opposed to having Docker spend 20 minutes downloading images and then crash because of a typo.

The problem is the bash script may end up depending on poorly understood aspects of the local setup (global config files, installed packages, etc) - it might work fine now, but then nobody runs it for 12 months and there’s some churn in personnel and suddenly people are trying to work out why it crashes. Dockerfiles can avoid some of that stuff, although not always (e.g. the common problem that if you don’t fix the versions of packages to be installed, an updated package is released which then breaks the Dockerfile)

[e_i_pi_2]

This is something we actively try to take advantage of at my company - we know that we've grown comfortable with architecture that may not make a lot of intuitive sense, so when we have new people join we try to make a list of confusing concepts so we can try to clean them up. In the ideal case the new person is able to do the cleanup, so we get a more intuitive design and they learn the surrounding architecture more along the way

[rr888]

The problem is when you join a high performing team and organization. Is a WTF something they should fix your you need to recalibrate what you think is normal?