Hacker News new | ask | show | jobs
by commandlinefan 1688 days ago
> Complexity-related Learned Helplessness

I've gotten used to just accepting that every job I have will involve a code base that uses up way too much memory to run in a debugger and doesn't have (or allow for) unit tests.

1 comments

There's a certain amount of "YOLO" fast-and-fragile development I'm willing to tolerate or negotiate. It's not like I have all the answers, and everything is a team effort.

But when I'm called upon to work entirely without tests, or with inadequate backups, or otherwise in ways that create a scenario where every keystroke I make holds imminent potential for disaster, I quit. I can do trapeze acrobatics if you let me use a net. If you insist that I must not use a net, I'm not willing to live with that constant stress.

> I quit

I did that early on. Then again, because the next place was the same. Then again, because the next place was the same. 10 places later, I finally realized they're all the same.

Huh? Like what specifically? All 4 cloud software places I’ve worked have had unit testing standards, cloud backups, even disaster recovery environments, etc. and none of these were mega corps. 50-1000 employees.
You can have those things in theory, but they need to adhere to certain quality standards to count on them in practice.
Parent said they visited 10 jobs that had no quality, I have 4 places that in practice try to have quality. So either they were hyperbolic, or has unrealistic standards, or bad luck, or bad job selection heuristics.
That's regrettable. Investing in reliability and redundancy is an optimization problem, resources are always limited, it's natural for there to be tension and I've grown accustomed to that. My experience has been varied, with some organizations investing more than others.
Maybe it's just me, but not having the net provides some level of "fun".

Most of the projects/companies/teams I've worked on didn't have tests, adequate backups or had scenarios where errant keystrokes blow up production but it's nothing that has been world-stopping and these jobs rarely involved anything were downtime equated to real harm (outside of the economics of the companies themselves, but usually nobody necessarily cared as long as things got fixed).

Companies that subscribe to YOLO-level development usually already _know_ that and with it, comes a certain understanding that shit will break, it's just minimizing the blast radius by learning/knowing the code bases and leaning on those with more experience.

YOLO-level development is a slippery slope... We had a case where 2 backends from 2 teams needed to interoperate.. to support a much talked-up UI/functional feature that was supposed to make our product standout from rivals. What was implemented was a rube-goldberg like contraption on the backend, where I dreaded being on-call as the talked-up UI feature was rolled out to more and more customers.

Oh yeah, we had unit tests all right, fragility didn't prevent that since you couldn't commit otherwise.

So there was no safety net. Just a constant dread of customers actually using the feature. So "fast-and-fragile development" resulted in a worst-case scenario as far as engineers were concerned.

The author says : "Their reports won’t tell the manager that they are having a terrible on-call experience". This was exactly what happened to me. And I quit one day, this was a big factor, two weeks out of a month (in my rotation) I'd be on the hook for supporting this morass of problems.

I will never understand this position. My life are already chaotic enough without factoring in the work-related part of it, it makes no sense to me why someone would actively invite chaos like the one you're describing. I'm much more excited by solving interesting problems or creating useful things than by having my work day interrupted by a production error that needs to be fixed immediately.
You're not necessarily inviting chaos, you're making a cost/benefit analysis of moving faster or safer. I prefer to move faster.

Moving safer takes up a lot of, imo, unnecessary time with dubious payoffs. I'd personally rather live on the side of YOLO and ship a lot then hem/haw over getting certain % unit test coverage (and the maintenance costs that come with that in the face of uncertain requirements).

> it's nothing that has been world-stopping

Well then you're talking about a completely different scenario than I am.

I could definitely YOLO out a change when I worked on code used by 100 people. When it was used by 10000, an error every now and then is forgivable. Now, if I make an error in a deployment I could stop 10,000,000 from using my systems. Similar outages have made newspapers. Even then, compared to FAANG scale I'm still playing with toys. The fastly outage was top story on the BBC while it was happening.
I would hope that world-stopping software (banks, critical infrastructure, healthcare, etc) have tests and all the things to prevent catastrophic error. For the rest of us, breaking things here and there in the name of moving faster is (anecdotally) more fun.
I think there exists something in between world-stopping as in healthcare (I want way more than just 'tests' there) and Facebook level (i.e. nobody really cares even though a lot of people will probably scream in "agony" when things don't work).

To maybe put it into examples:

    World stopping: Traffic collision avoidance system (TCAS) has a bug that lets planes crash into each other.
    Don't care: Not enough vegetarian meals made it onto the plane because of a bug in the meal reservation software.
    In between: Thousands upon thousands of people get stranded in some airport for 3 days because of a bug in the route planning software.
This was an example within the airline industry but I'd say there are many more examples of where enough people depend on something for things that do objectively matter enough to warrant "good enough" software and software development practices that don't result in large-ish outages that then need to be fixed. Inconveniences I'll take because we all aren't perfect, even with testing.
> healthcare

most of healthcare software (that i have worked with) is really, really bad.

The fact that it mostly has no tests, or if it has tests they mainly test just the happy route.

They usually have lots of documentation, but it's not comprehensive and mostly just describes the happy paths, so when thing break you are out of luck. Also can't google anything since this are all closed source enterprise type software, so you are at the mercy of their support, that often also sucks.

And if you have to integrate with it, you often can't get the documentation, or tools to do it unless you are working for one of their VARs. And the VAR's are often even worse.

If you pick random 10 high school kids, get them into some code academy for a 6 months, and then have them develop an application, the ones that finish will probably be better than majority of health care software out there (for simple reason that they won't know enough to really screw up as bad.).

The only saving grace is, that its often written in Java so you can find workarounds in runtime (or inject stuff) if you really need to.

The fun should come from solving fundamentally difficult problems, not from someone fucking up a CRUD response because they didn't properly test their filter parsing.
Lots of companies have no "fundamentally difficult problems" when it comes to their software. Lots of software developers don't have any of these problems to solve.

(I'm also against this "YOLO because it's fun" attitude, though.)

If this is the case you still have a fundamentally difficult problem, it's just become something like "how can we keep our velocity while also improving test coverage / reliability?" and the solution is usually some new internal tooling.