Hacker News new | ask | show | jobs
by itsanaccount 532 days ago
I would like to register a prediction of futility here, just a little more detailed than the trope of "when you try to idiot proof something the universe always makes a better idiot."

The number of software systems I've seen designed to allow the clueless to bumble their way through operations is much higher than the number of companies I've seen that attempts to train better operators.

Maybe its the experience of working on tractors, where the PTO has fun side effects like "degloving" where, sure theres a guard but more importantly theres attention not to put your hand, hair or clothing near the spinning torque monster of doom. Theres no option to make that purely safe economically. Same goes for machine tools and other heavy equipment. The people who work around such things have been to my experience more capable at accomplishing difficult, diverse tasks.

Where was the last company that had anything equivalent to the lowly "forklift certified" for prod? Its a very rare shop I've seen invest in any sort of across the board training for command line skills, outages, pre-requirements for delicate operations. We don't invest in people being better, being more capable. I think because we have internalized an owner/management point of view that workers are fungible and training is a waste, while software system guards are investments.

As a worker, I don't agree with that. I don't agree with building systems to be powered by the lowest educated, lowest paid meat popsicles yet I think thats the strategy behind this.

Next time you go to build a system like this, consider who's logged into production and can they be trained to be more capable, more attentive operators. In the long run, I think it will end up with a better industry overall.

5 comments

To be fair, the specific use case mentioned on the github is typing "shutdown -h now" into the wrong terminal window, and the way it's solved is quite clever: it asks you to enter the hostname of the system you're actually trying to shutdown.

This is something that could conceivably happen to people who are properly trained. This also means it doesn't fall prey to the usual "do you really want to do this? yes - no" prompt, where you just get used to automatically hitting "yes." Even if you habitually enter the hostname, it'll be wrong if you execute the command in the wrong terminal.

I have no doubt this would stop some instances of (say) accidentally rebooting a production server.

But as an idiot, I’m here to tell you it isn’t idiot proof. In a moment of suboptimal attention, I could easily ask myself “What host am I on?” rather than the more appropriate question “What host am I trying to shutdown?”

In my case it may be quite a bit worse because I have hostnames in PS1 specifically to avoid running any command on the wrong host. With the hostname right in front of me I could easily accidentally habitualize just typing the hostname I see.

Over time, I imagine this would befome more likely rather than less, so although I think it probably helps, I am sympathetic to GP’s view of futility.

Maybe have it prompt: what host do you NOT want to shut down?

/s

Framing it as an "idiots vs not idiots" is the wrong way to think about it.

When you are working with the PTO you know you are working with the PTO. You are standing next to the tractor. The PTO should always be handled with the understanding that it can be dangerous. Here people are doing something mundane, everyday and not dangerous (shutting down their own laptop) and suddenly it becomes dangerous because they mixed up the terminal they typed their command in.

It is as if sometimes your alarm clock would be replaced with a PTO. Your alarm beeps, you reach out to turn it off as always and bam your hand is gone. That is the situation we are talking about here.

> I think because we have internalized an owner/management point of view that workers are fungible and training is a waste

This is not something you can train out of people. Your best trained, and most skillful operator can do this mixup. This is not that someone doesn't know what "sudo shutdown -h now" does. They do very much know it. Like the palm of their hand. What they don't notice is that they are typing it into the wrong terminal.

Weirdly enough, I have thought about this and to a degree, this can be interpreted as normalization of deviancy and normalization of danger.

Like, yes, I am a console jockey and prefer working in shells with a tiling window manager, keyboard only control and such. I will however always shutdown my workstation with the mouse (or trackball rather) through some UI of the desktop manager, or a desktop manager specific way.

Otherwise, I am normalizing the use of sla-dangerous commands like "shutdown". It takes that little bit of fear and respect out of those commands if you use them daily for no good reason or if better choices exist. Like, don't turn your alarm off by cutting it's wire and re-soldering it later.

And similar, if I need to reboot production systems, I'll much rather reach to some control interface of the virtualization, or use something like ansible to dry-run these dangerous tasks first.

Or it terrifies me how care-free some people are with "sudo rm -rf". I've caused myself so much pain with rm. "sudo -u app-user rm -rf" is right there, or even better, "sudo -u app-user find -name foo -print > stuff; cat stuff" and later some "xargs -i rm < stuff" and "xargs -v rm < stuff". Yes it takes a minute more to do, but it prevents ... accidents.

It’s the “Do you want to share XXX” popup all over again. Once you get used to see it, it will be a part of the shutdown command (to take the example in the readme).
Hahaha, yes I thought of it while writting it! This is a crazy annoying terminal popup! But it can maybe be useful if used correctly and with parsimony
As a counterpoint, aircraft designers have learnt through bitter experience that interface design is extremely important, as in stressful situations, or even due to a momentary lapse in attention, even a very highly trained, careful individual can make a fatal mistake. Quite frankly the only reason tractors don't get the same treatment is that generally the operator only kills themselves, not a few hundred other people.

(This isn't to say that training is useless, just that more training isn't the best and only solution to all problems, nor will a lack of careful interface design magically create more capable operators)

I think that's a great counterpoint, but it is leading to the same issue. Its allowing a class of pilots to fly who know less and less, and are more reliant on the automation. With deadly results.

Separately the only reason tractors don't get the same treatment is because society doesn't care about rural men in the same way they don't care about soldiers. In comparison cars have been largely regulated for safety, because the people who die in car crashes come from a wider swath of society.

> Its allowing a class of pilots to fly who know less and less, and are more reliant on the automation. With deadly results.

This is a plausible hypothesis, but it is reflected in the data? Flying has gotten safer and safer over the years, but of course that's got a multitude of effects contributing to it, not just the skill of the pilots. Reliance on automation is effectively a requirement for modern aircraft given the number of control systems which are critical to the pilot having any control of the aircraft. I've seen Boeing criticized for their approach here: while Airbus's interface is more or less "you are directing a series of control systems, not flying the plane directly", Boeing has essentially tried to concoct an elaborate illusion that a gigantic airliner is a Cessna, which a leaky abstraction even if it makes the pilot feel like they are "closer to the metal". (I could draw a comparison to C programmers who feel the same thing despite the great honking illusion of an optimizing compiler in between).

> The number of software systems I've seen designed to allow the clueless to bumble their way through operations is much higher than the number of companies I've seen that attempts to train better operators.

I agree, and that's a problem. Now with LLM, we're training everybody to not know how to find information by themselves (we call that "prompt engineering" so it cannot be bad, right? /s).