[terminado]

The Ivory Tower has a way of phrasing concepts such that they are framed by finality and totality. The mentalility comes with having made it through the admissions process, and passing your final exams with a good grade point average, conferring some lofty degree of distinguishment into one's possession.

So, to look at the words:

  The notion that software engineers are not 
  responsible for things that go wrong will be 
  put to rest for good.

I'd have to say that this sort of high-minded platonic concept needs some revision.

  The notion that *some* software engineers *cannot*
  be found as responsible (in part or in whole)
  for *some* things that go wrong will be 
  put to rest in *some* situations.

There needs to be a degree of responsibility ascribed to some classes of systems development.

Meanwhile, there is very obviously a line to be drawn between the programmer that programs their VCR clock to time a recording, the programmer that programmed the VCR as a consumer-grade product intended for purchase by unlicensed individuals, the TV network that broadcast the television show at the time the individual programmed their VCR to record 60 minutes of broadcast on a given channel, and the programmer who locked me out of the firmware on my smart phone.

[goalieca]

> software engineers

I've had the idea for a while that most of us practice software development rather than software engineering. I have a degree in computer engineering but i consider myself software developer now rather than a software engineer. The reason, I don't practice engineering in the legal and professional sense.

In engineering school we learn about engineering as a formal process and professional responsibility. Both of these things are largely absent in most shops now. I get that not all projects need to be professionally engineered with all the costs and timelines associated with it. I think this is why agile came along. Sometimes it's just good enough to hack something together and demo it until a manager says it's time to release.

But there are many other projects which are extremely important to society and should follow more traditional engineering practices. There shall be external and internal engineers who must formally approve any product before release. There shall be specific and testable formal requirements. There shall be a formal design and documentation for engineers to review and people to develop from. etc. etc.

[terminado]

There's no legal framework, to distinguish devices that matter from devices that don't.

There's no clear demarcation between devising a convenient contraption, versus implementing an inadvisable hack that leads to a hazardous outcome, especially within the scope of web based systems, since no portion of the internet is to be regarded as reliable life-saving infrastructure.

I think the mistake is to trust packet-switched networks and peer-oriented protocols as reliable systems at all.

If you cannot control the whole system, end-to-end, and any unwitting peer can over-consume bandwidth (jamming traffic and communication with interference), effectively cutting you off from a necessity, why would you bet your life on the availability of that system?

[transpute]

Work is underway to support Time-Sensitive Networking in modern operating systems: https://schd.ws/hosted_files/elciotna18/b6/ELC-2018-USA-TSNo...

[transpute]

An effort was started by Mudge to use static analysis and fuzzing to assess a range of software, https://34c3.cyber-itl.org & https://theintercept.com/2016/07/29/a-famed-hacker-is-gradin...

"... first-of-its-kind method for testing and scoring the security of software — a method inspired partly by Underwriters Laboratories, that century-old entity responsible for the familiar circled UL seal that tells you your toaster and hair dryer have been tested for safety and won’t burst into flames. Called the Cyber Independent Testing Lab, the Zatkos’ operation won’t tell you if your software is literally incendiary, but it will give you a way to comparison-shop browsers, applications, and antivirus products according to how hardened they are against attack. It may also push software makers to improve their code to avoid a low score and remain competitive."

[downer68]

This is a flawed concept from the outset. You either have appliances, or computational platforms.

Turing-complete systems are arbitrarily flexible as a matter of principle.

If the firmware can be altered; if any addressable memory can be changed, and a system relies on an internet connection for maintenance and support, it is an unreliable system.

[transpute]

Their site mentions relative assessments of different products, not absolute claims:

"These sorts of questions can’t be answered in an automated fashion, due to theoretical obstructions ("undecidability") first identified by Alan Turing. Thus, to measure security in a practical fashion, we employ heuristics. We don’t need to find any specific vulnerabilities in order to assess how secure software is. Instead, we can observe the software’s safety features, build quality, complexity, and other heuristics. Some heuristics directly impact software security, while others might just be properties of software that are generally only found in cases where development teams know what they’re doing. As long as they correlate, it doesn’t matter."

[downer68]

Sounds like stop-and-frisk TSA pat-down security theater. This is not the kind of concept that I'd ever expect to withstand a determined adversary.

Especially for machine learning scenarios, don't expect a show of force to prove as an adequate deterent. Posture and presumptive correctness aren't enough to protect you from entities ignorant of fear and indifferent to wastefulness.

[transpute]

Thanks for the astute observation.

It's not clear that their goal is deterrence. If they are transparent with heuristics for ranking vendors, it could provide fabled market-based incentives for a vendor race to the top, narrowing the security gap between "best" and "worst" vendors. If highly rated vendors advertise their achievement, buyers could factor the rating into purchase decisions. The heuristics would need to evolve as the floor of vendors' security practices is raised.

The bigger problem is what happens if the market-based approach fails? Will regulators step in for certain classes of software? Regulators are less likely to understand Turing-completeness.

[downer68]

Considering the topic of this thread, the debate is fundamentally about automation and machines operating with the authority to make decisions.

There are at least four broad, complicated areas of effect that need to be addressed to begin grasp the scope of what's at stake here.

  1. Hardware interfaces for software control.

This is the entry point that puts lives in harm's way. Software can do whatever it wants to a terabyte of RAM, but there's no outcome, if that RAM isn't driving a real world system. This is the point at which the tires spin and the steering wheel turns. Machine Learning is mostly about probability and statistical liklihood, so good luck unit testing that.

  2. Decision making and autonomy.

This is where, after the system comes online, checks its supplies, and determines it has everything it needs to attempt an action, it decides to do something. After Windows boots up, the autonomous entity sits at the keyboard, checks for disk space, battery life, time of day, gas in the tank, oil pressure, and anti-feeeze, and then considers options for where to go for a drive. Perhaps the corporation that owns the entity configures a default strategy of taxi service, before hauling cargo from a mine to a warehouse, or trash pickup assistance. Taxi service reduces wear on the vehicle chasis, so the entity opts to drive as a taxi. It checks in with a ride-hailing service, and offers it's resources to the pool, and queues up for an assignment. Since we're talking about an autonomous system, there's a division of activity between deciding to try something, versus carrying out the behavior that effects outcomes. This phase is simply parsing resources and allocating them, but not using them to do a thing. Budgeting capacity is a different form of autonomy than performing specialized work.

  3. Succeeding at a discrete task.

This is where the car negotiates the entire trip from garage to destination and back, without damaging property, harming others, getting stranded, and hopefully turning a profit, or at least fulfilling it's role as an appliance with a warranty (90 days, one year, 5 years or 100,000 miles?). This may be vertically dependent on multiple sub-products originating from multiple corporations, all interacting to produce the talent or skill exhibited by the system. Maybe it's a smart refrigerator, and it's defaulted to always keep a gallon of fresh milk available, but you never drink milk, and you don't know how to turn the feature off. Who's to blame for all that spilt milk? Maybe your autonomous taxi can deliver milk to the houses without smart refrigerators. Who's to say it's a problem or not? Maybe the self driving vehicle wouldn't have run over than pedestrian if you could have convinced the fridge to stop replenishing the milk. Which brings us to...

  4. Economic realities, sociological effects, 
     propaganda and psychological operations.

Yeah, great. Your car can drive itself around 24/7 earning a passive income for you while you surf the internet, and bandy about more great ideas. You and 20 million other people are all doing the same thing. This has a transformative effect in aggregate. Your car works as advertised. No bugs. No accidents. No injuries. A perfect driving record. But there could be deeper sociological ramifications to the introduction of such technology that is unanticipated, and thus hard to envision. Did Facebook drastically augment the outcome of an election, if not willfully then perhaps by sin of omission? Would we have imagined such a conversation in 1999, before the dot com bust? Why didn't AOL produce this sort of acrimony? So, if uber's gig economy and side hustle isn't disruptive, while we work out the kinks that run over pedestrians, what about the decentralized mastodon for self driving side hustles? What about self driving transoceanic zeppelins, over international waters, than launch their own weather satellites? Who will stop them from polluting low earth orbit?

Some of this isn't about software bugs. Some of it is behavior and psychology.

[Latteland]

That comment about admission to and completion of college leads to simplistic conclusions comes across as unfair. Pretty much every educated person went through such a sequence to get through college (others are educated without college, but they are tiny minority...). And plenty of us are able to understand the world has nuance.

ON your latter point, I agree that there needs to be a degree of responsibility to some software dev, but it's a complex area and I can't really organize 'blame'. What's the blame I should get for writing a better webserver that some dicator uses to serve up his orders and have people killed? Compare that to the firmware of a phone programmer, and the person who teaches a robot how to determine if someone is killed by a weapon. It's too easy to look to that last person and say without thinking that they are the bad one.