Hacker News new | ask | show | jobs
by ISO-morphism 1152 days ago
I believe that the engineers are competent at developing software, and think that a lot of the quality issues are due to engineers abdicating authority over the software to managers and never saying "No, adding in that feature at this point in the timeline is going to make this a steaming pile of shit and I refuse to do it and still call myself a software engineer."

I get that we have families to feed, but I've seen far too much of a mindset shift in fellow engineers into thinking that we're warcraft peons rather than professionals. "The business" has engineering feedback as a necessary input, and speaking individually with steakholders they expect this - they'll push until we push back.

9 comments

There's no incentive for an engineer to do that. Saying yes and delivering crap gets you a bonus, hard truths get you shuffled around or made redundant. There's no real consequence for delivering crap, so that's what happens.

Contrast this with other engineering fields, where the engineer is truly responsible for the decisions they make. My civil engineer friends face losing their licenses, fines or jail time if they are found professionally negligent. The same is true of other high stakes professions - think doctors, lawyers, even accountants. It's probably not appropriate for most software engineering roles, but for safety critical systems it doesn't seem far-fetched to me.

The product team for the touchscreen control system scoffs at the engineering team’s concerns because “customers don’t care, they’re wowed by the touchscreen at the dealer lot.” It’s only after purchase that the regret sets in. Product teams know this and exploit it. The business side knows they’re selling a steaming pile to customers and don’t really care for engineering’s concerns. In most situations they’ll override these concerns forcefully. It’s a hard pill to swallow as an engineer in these companies.
"My civil engineer friends face losing their licenses, fines or jail time if they are found professionally negligent."

These standards ought to be applied more widely and done in conjunction with tightened consumer law. In many cases the quality of electronics equipment has gone to the dogs. I could give instances of appliances I use that can only be used in a hobbled mode—numbers of published functions simply don't work—because their firmware bugs are so bad.

These devices are so bad they wouldn't pass as early developmental mockups let alone early prototypes in a professional engineering establishment. I'm damned if I know why the hell consumers put up with the situation and haven't revolted, it remains a mystery.

Things won't improve until they do.

I don't think jail time for making software too bloated and slow for your liking is a serious proposal.
I'm not a strong advocate of this, or of jail time in general for non-violent offenders, but as a thought experiment, suppose that Acme Auto release updates to their car's software which make the UI more laggy and less intuitive to navigate. After they do, there are a cluster of similar accidents - distracted driver hits a pedestrian when they should have stopped. These can be shown statistically to affect Acme models with the software updates significantly more than any other make of car, and more than Acmes which don't have the update. A class action lawsuit is started against Acme by both crash victims and drivers. In discovery, correspondence between software engineers is found. Engineer A writes to Product Manager B and says that they don't think the new build is safe, because they were forced to compromise latency performance, and button placement is now more surprising, having changed again. QA Engineer C chimes in and says that since the changes apply to features critical to driving such as de-misting, they won't be prepared to sign off on the change. PM B says that they have to go with the new version in order to meet internal targets on engagement with entertainment apps. They overrule A and C, as company rules allow them to do.

Do you think B should face any personal consequences within a public justice system? Or Acme is just liable for a big payout and then upper management decide who takes the blame?

"Do you think B should face any personal consequences within a public justice system?

Yes he should. Reason, because he now knows the consequences of proceeding if the problem not fixed first (he was told them by engineer A.).

Once aware, everyone has the responsibility to act. The Occupational Health & Safety laws of many jurisdictions are written exactly on this principle. Such laws don't just apply to managers and decision-makers, a floor sweeper who overheard the conversion would also be culpable if it were proven that he did not inform authority of the fact and or if he had good reason to suspect Management would do nothing.

Same for Engineer A, he would still be culpable if after telling Product Manager B the facts and he knew or had good reason to suspect Product Manager B or others responsible did not or would not act to fix the problem. Moreover, unlike the floor sweeper, Engineer A, due to his extensive knowledge of the facts and his senior decision-making position (as an engineer—even if not in charge of marketing or production), the Law would still require him to follow though with either senior management and or external authority until he was satisfied (to the level of his professional ability) that the problem was sufficiently in the hands of responsible others.

Whilst these laws vary between different jurisdictions the common themes are if one—and that's anyone, inside or outside the company—knows there's danger and or potential for someone to be harmed or killed then that person has to act, irrespective—full stop. Second, the more responsible or more knowledgeable someone is as to the consequences of something or some process going wrong then the more incumbent it is for that person to act (the floor sweeper in Boeing's factory would not be expected to know the wrong alloy had been used in engine turbine blades but the engineer would).

These laws were introduced to avoid problems like the Challenger and the Boeing 737Max disasters, and the Purdue Pharmaceuticals opioid crisis. Unfortunately, the US lags behind in either implementing them or making existing laws sufficiently strong.

> I don't think jail time for making software too bloated and slow for your liking is a serious proposal.

Especially since the boundaries of "making software" is pretty blurry.

Would creating a suboptimal Excel spreadsheet count as "making software too bloated"? A pretty strong case could be made for that.

Would creating a clunky personal homepage count as "making software too bloated"? A pretty strong case could be made for that.

You don't find the idea both kinda hilarious and somehow vaguely appealing even though it's a bit nonserious at the same time? I love it as a thought experiment.
Ideas that are silly thought experiments have a way of actually getting implemented once too many people start paying attention to them.
"too bloated and slow for your liking"

I neither said nor implied this. To be clear, the products in question were sold under false presences, as they were sold with features that—as far as the lay consumer is concerned—don't exist (just because I'm a technical person and I know they are almost certainly software bugs and or are not designed as per specifications is immaterial). In essence, by deliberately selling a substandard product they've committed fraud.

Here's one of the many examples I could list but it's a clearcut easy one to understand. I have three identical PVRs/STBs (Personal Video Recorders/Set Top Boxes) of one brand and type—so the problem is not just a single faulty unit. These are the type that you add external storage via USB, 2.5" or thumb drives.

Advertised on the outside of their boxes is the statement that they will take external storage to 2TB in size, the scanty manual—if you can call it that—that's sealed in the box which you can't read until one unboxes the device makes a very clearcut statement that the maximum limit of external storage is ONLY 700MB drive (a rather strange limit methinks), and 1/3rd that published on the box. In practice, these units simply wiil not work with ANY external USB drive 2.5" rotary or SSD drives—even the lowest current SSDs of 120GB or smaller—which is in direct contradiction to what's stated on the box and in the so-called manual.

They will however work with thumb drives up to 128GB (I haven't tried bigger). Incidentally, have you ever seen a 2TB thumb drive? Right, I haven't either.

Thats not all, there are software bugs and an UNSTATED limitation that only six programs can be programmed at one time (this is an unheard of restrictive limit, I've never reached the limit on my other units although one type, which has other bugs and problems, says its limit is 32).

I also have three other PVRs but of a different brand (a well-known international mob). All three have the SAME identical model number but two have completely different electronics and their firmware operates in a totally different fashion to the first (clearly built by a different subcontractor), Even the boxes they came in are all identical.

I discovered this when the first unit failed and I bought two more of the same. Moreover, the first unit wasn't even out of warranty so the second purchase was only about six months on from the first.

To make matters worse, before the first unit failed and after getting nowhere with the local distributer I'd hunted around the internet for a firmware upgrade to fix the annoying bugs but couldn't find an ungrade (little wonder if different hardware exists for a given model). The so-called identical replacements are not only operationally very different but they have so many bugs that they are actually unusable. I'm still working on exchange/warranties and such.

Those two brands are not alone in having masses of bugs, I've three other brands—five all up with even more model numbers (yes, I've boxes of these damned things). The bugs in a third band are so bad that it allows one to program the same timeslot on different channels simultaneously—which channel takes precedent and is recorded is pot luck, at other times, about one in three, it fails to record the scheduled program, only a black screen (it switches to blank instead of a channel—but give credit where credit's due, it does switch to blank at the correct time)!

And believe it or not that brand/model has been on the market for several years and it still is without any firmware upgrages being available.

Here, I've presented only the tip of the iceberg—and that's only the PVR/STB story. Where else would you like me to start?

People should not have to put up with this shit, it wastes time and human effort not to mention wastes resources and the environment is clogged up with dead and discarded e-waste and other junk. A simple way around the problem would be to license both companies and their design engineers and threaten them with loss of license for producing junk. With importers, bring in junk and they'd lose their import license.

Implement these rules and most of the problems would soon disappear. In extreme cases where irresponsible designs threaten safety and life then loss of license and jail time would be a just measure.

You'd just get a lot less software as people instituted enough checks to make progress glacial. Not everything needs to be developed like it is a medical device or aerospace software.
Singapore has criminal liability for software malfunctions. I don't think they've sent anyone to jail for a software bug yet, but the law allows for it.
Rightly so, if justified by the consequences—to the extent of causing injury or death.

As with other professions, civil, chemical engineering etc., when the outcomes are the same (people killed or injured etc.) then the punishment should also be the same.

Software design should be no exception to any other profession just because it's common for programs to have bugs.

Moreover, the profession of programming now calls itself Software Engeering, if it wants to play with the Big Boys then it must face the same consequences when things go wrong.

The incentive is having a rewarding job where you develop products you are proud of. Once I have food to eat, this is by far the most important incentive for me and it greatly outweighs e.g my desire for promotions, raises and bonuses. If I can have both, great. If I need to choose one, it’s the fulfilling job and product pride every day.
No thanks. I’ve got a family to support for at least 18 more years, and I could be laid off at any second.

I need to make as much money as possible in the little time as possible, and the best way to do this is to stop worrying and learn to love the bomb.

kinda weird that expressing pride in one's job and prioritizing excellence over financial remuneration would be downvoted so vigorously
Yep agreed. If you raise a flag, you'll be looking for work. Head down and build crap, and you have a job for life. I see it all the time. I've lived it.

Sometimes you have to decide, do I build a better system, or do I feed my family. The craft and world suffer, but...

It should be, at least on countries where Software Engineering actually means something, and not a title that one can easy peasy call themselves after a six weeks bootcamp.
Oh this again. Yeah a certification is going to solve all the problems.
It will, when it comes with a liability just like any other Engineering position.
I think it's the liability that matters, not the certification -- which usually translates to "X years in a government facility, pretending to learn something which may or may not be misguided and out of date."
I have a 2 month old baby, and got fired from my last job after 6 months of unpaid overtime...

I am currently trying (again) to get into embedded development. I would gladly take the job of the guy that refused to make shitty software, if that means I can keep feeding my baby. Unless it would be too unethical. (I refuse to do work that will kill people, for example I won't work for Palantir, companies that make sketchy software related to flight controls or medicine or other critical applications and so on... but the car media player? yeah, I am willing to make a crappy one if I get the job, I prefer to make a good one, but if my boss want a crappy one... then what I can do? overtime to get a promotion, clearly doesn't work ;) )

> I refuse to do work that will kill people

Isn’t a crappy UI on a car something that could kill people?

In some abstract sense, like how you could say that you should absolutely not drive a car as you could accidentally kill a pedestrian.
I don't think it's all that abstract.

We know that phone using drivers perform worse than drink drivers. They regularly kill people because they're not concentrating on what's in front of them.

If you change the car's UI from something with low latency[1] to something with much greater latency[2] then you are definitely putting others at higher risk because drivers spend longer not concentrating on driving.

--

1. See button, move hand, feel large physical button, look back at road, press button, feel feedback click.

2. See screen, move hand, see screen pops up menu on hand proximity, see menu item, click menu item, miss-press try again, wait for animation, attempt to select feature but hand moves due to bump in the road, move hand again, try to select feature again, miss-press try again, wait for feedback animation, look back at road.

Then why is anyone skating for the decision to abandon buttons in the first place? The driver is responsible for paying attention to the road rather than fiddling with the radio.
> Then why is anyone skating for the decision to abandon buttons in the first place?

Cost.

My guess would be that manufacturers want touchscreens because they're cheaper to develop and implement than an inventory of individual physical controls.

Adding a new touchscreen widget to a car that's already in production is just an over the air software upgrade vs a very expensive redesign/recall for physical controls.

Drivers don't think through the consequences of the control system at the time of purchase or have it as a low priority compared to things like purchase price.

I think it's pretty easy to tell yourself that your not going to mess with a screen while driving but in reality it's much harder to fight that compulsion. If it wasn't then we wouldn't need the "I'm Not Driving" feature on phones.

We completely understand - and we would all make the same decision.

Which is why (in the UK) I view the rise of strikes and unions positively - workers need to hang together - set standards that they won't go below, and also ensure the spoils are fairly shared out.

Recently the deputy governor of the bank of england said "inflation has made everyone poorer - get used to it not ask for wage increases". Which tone deafness misses the point that this is not about wanting a 10% pay increase - it's about how is society structured and how do we share the vast wealth.

The question is not how do I get more, it's how do I build a fair system that inwill be part of.

we are playing the sociopaths game as if it's the only game in town. There should be no game we all play where the rules include not feeding babies. That's a bad game

>"inflation has made everyone poorer - get used to it not ask for wage increases"

Which is absolutely horseshit, because the European countries that had the least amount of inflation (besides the super wealthy Switzerland and Luxembourg), were the countries where salaries are automatically indexed with inflation, like Belgium.

So it turns out that forcing companies to increase wages by inflation is a very effective deterrent against price gouging and inflation, yet those in charge try to convince us otherwise, that we should accept lower wages for our own sake lol.

Can you believe the nerve of these people? It's disgusting, but we get what we vote for. You want better, then vote better.

I did not realise there were any countries doing index linked salaries.

Nice idea

> You want better, then vote better.

This doesn't help. Even if you were the guy that read all policies or whatever a candidate pushes, the vast majority if people are still just gonna vote with their emotions.

The industry works as follows: the OEM (Daimler, Toyota etc.) says they want a software spec implemented. Other companies (Tier 1 suppliers) bid to win the contract. The cheapest usually wins.

If you have a company and refuse to take the crazy deadlines and low quality and low pay then don't worry, there is another Tier 1 supplier across the street who will do it for you. OEMs know that you need them more than they need you.

And on top off that, some product manager at Mercedes might say they absolutely need let's say Atmos in the next release and ask the Tier 1 to implement it. The Tier 1 usually uses multiple Tier 2s. The Tier 2s say they need more memory, but that's not practical at all because that hardware was fully validated years ago and you just don't make incremental changes to automotive hardware, and there won't be a hardware refresh for another 4 years. It is in none of the tiers' interests to say they can't do it and lose out on a multi-year contract so they do the best possible job within the constraints.
> Atmos

Well, that is evil by design irrespective of the particular implementation, no? ;-)

That’s fine. But the worst case response time across the entire UI must be the first item on that spec! That spec is of course the responsibility of the OEM to create. Where this goes wrong I don’t know but competition, price sensitivity etc doesn’t explain it. Having soft and hard limits for response time seems obvious and someone either forgot, or they had a meeting where they (the OEM, after deciding on a solution and having it implemented) said “ok we can save $20 on the BOM by going for a cheaper SoC if we accept 200ms response times instead of 50ms” and that should basically be criminal due to the safety aspect.
It's not criminal so they will continue to lower the cost. 50ms quickly becomes 500ms response time when you realize that no tier 1 supplier can hit within your OEM budget and your boss is becoming impatient :)

Another thing that happens is when there is a 50ms response time mandated and at the start of the project you have 30ms and it slowly creeps up until you get to 55ms. Then the blame game starts. Like I said previously, each component is usually won by a different tier 1 supplier. So everybody puts the blame on somebody else. One tier 1 is building the linux distro, the other is making basic system libraries and the applications are written by another 20 tier 1 suppliers.

Then the deadline hits and the car needs to be sent to the showrooms. So whatever 50ms was chosen initially is changed to whatever it currently is so to say they are within spec and can ship the damned car.

I think this describes a lot of how software is developed (which just goes to show how immature that part of the industry is.). Similar contracts and agreements exist between manufacturers of the brake system as well but if there is an issue there where the ABS system doesn't prevent lockup in 1% of cases, then the problem is solved or the car launch is delayed, because that's what the system is designed to do. The problem here of course is that these things are seen as gadgets without safety aspects and there is no regulatory oversight.

I guess it's also up to buyers: don't buy cars with shoddy infotainment.

ah, there is some regulatory oversight and there's awareness of the safety aspects.

Not much, but, there is some.

After working on infotainment, I'm doing my best to avoid purchasing a car with an infotainment system. I had to compromise and got a Civic with an infotainment system for my wife, but I'll probably stick to late 2000s, early 2010s for as long as I possibly can to avoid them. Seriously hate touchscreens, and there's just not enough value in an infotainment system for me to be OK with what I'm losing by having one.

Something missing here is that the Tier 1 supplier will take one look at the spec and know that latency hasn't been mentioned or considered.

They will not tell the customer this because either the customer doesn't care, so why waste time and resources

...or when the customer realises they will open a Change Request to fix the issue. Change Requests are how you make an actual profit on unprofitable, low-balled contract and probably gain an extension on the unachievable timeline agreed to win the bid in the first place.

The "customer" is the person that signs the contract not the user of the product. They probably don't care that it's a pig to use. It just has to look OK in a presentation to their boss (...who isn't going to ever use it either).

If Tier 1 suppliers didn't behave this way they wouldn't be able to pitch bid responses cheap enough to win bids. The responsibility for crappy products lies entirely with the product owners. Only they know what's good enough.

From what I observed product owners know that the system sucks and it's laggy. They also know that the OEM management only keeps repeating "cheap, cheap, cheap", "with a renewed focust on costs" and other sayings like these.
That was true and it might be true for most of the production process, but as you may read in this article https://arstechnica.com/cars/2023/02/mercedes-ceo-tells-ars-... they are actively working to change and speed up the software development process:

Källenius explained the current system to Ars Technica like this:

    "So some engineer in Sindelfingen comes up with a concept. You have to write that down. You have to send it to the supplier that needs to be quoted. Then procurement people need to negotiate with each other. Then that supplier goes to some sub-supplier in Eastern Europe and wherever they do. It goes back up the chain again. It gets tested and nine months later, you have actually changed something in your infotainment system. Now you go into the ESH [Mercedes' electric Software Hub]. To say, let's change this and you just do it."
Working on a product vs working on a project. The latter means that the usefulness of the thing produced is a non-goal because the customer is not a user.
It's very rarely a developer's role to say outright "no". It's our role to make trade-offs understandable by decision-makers, and to clearly articulate why we think something is a bad idea. Sometimes (usually...) the broader business has different goals that just delivering quality software, and I think part of being a professional is understanding that.

There is, of course, a time and a place for a hard "no". I've genuinely threatened to quit rather than implement a particularly user-hostile feature in the past.

I completely agree with you, and I think you've worded this better than I have.

I was trying to advocate for the middle path, where there's healthy communication from both sides, to the point that developers trust when "the business" makes a decision - not to the point of engineering completely blocking the, unquoted, business.

Yep. It's part of your job to tell managers "yes, it's possible, but it'll negatively affect performance in a significant way because of such and such hardware limitations, so it's not the most valuable feature to implement at this time".
And it's part of the manager's job to tell you "I agree but don't care. Do it anyway or I will find somebody else to do it. For cheaper." Always remember that you are replaceable.

When the newborn is crying and wants food you do what you've got to do. And there are many people like that.

Nah, that sounds like bad management. They should be relaying the requirements to the hardware team.
Most of the time the hardware team doesn't get to choose their hardware. There is usually a budget and in automotive you can select only hardware that has some specific certifications for use in a car so that narrows the list even further.
And once that hardware is selected for a given range of vehicles, it's literally impossible to change. Updates to some parts of the software stack are also basically impossible at a certain point.
It’s your job to tell them if the requirements cannot be met and that something’s gotta give. It’s his job to decide what, and it looks like he’s choosing latency When push comes to shove.
I don't own the company. I will push back a bit, but if management insists on using the foot-gun, why would I get on their bad side? They are adults as well and if they don't agree with me, I do as they tell me.
I often see sentiment like this here but I think there's a disconnect between the sacrifice you're expecting people to make and harm caused by the work. It's admirable for someone to put their own job security second to obeying really harmful directives, but something that's going to make the music system laggier?
Those managers might be right from a sales point of view. Until we buy all this, they are right.

Let’s see that Tesla can change this or not.

Nope, it's too crude a metric for such features
Work work.