[daenz]

>Right now, most of what the computers in the world do is to execute tasks we basically initiate. But increasingly our world is going to involve computers autonomously interacting with each other, according to computational contracts. Once something happens in the world—some computational fact is established—we’ll quickly see cascades of computational contracts executing. And there’ll be all sorts of complicated intrinsic randomness in the interactions of different computational acts.

I don't think it takes a math genius to see how this is a bad idea. In the same way that trading algorithms can get into a feedback loop that crash markets, these "computational contracts" can cause cascading failures that hurt society as a whole. This is why human intelligence is so critical in running society: it has the ability to question whether its "programming" is correct and having the intended effects, and adjust accordingly. Computational contracts have no such introspection, by definition. They resolve because the rules are satisfied, for better or worse.

And all of that isn't even considering the attack surface area for malicious actors to target.

[raz32dust]

In the same vein, "Human intelligence" led to the 2008 crash. Human's aren't infallible either. My problem with this argument is that there is no logical basis for how exactly human intelligence is different. Computers aren't there yet in most domains, but there is no evidence to say they can't get there. How soon is anybody's guess right now.

While AGI might be far off, I can certainly imagine computers running larger and larger sub-worlds. e.g, if all cars were self-driving, I am reasonably sure we can design traffic to be more efficient with all cars coordinating with each other instead of humans trying to do so.

[jerf]

"My problem with this argument is that there is no logical basis for how exactly human intelligence is different."

I can give you a partial answer, which is that human intelligence is somewhat slower, which mitigates the ability to crash the entire economy in 15 seconds. That's why a stock market can crash in 15 seconds nowadays, but "the housing market" can't. This gives people some time to do some things about it with some degree of thought, rather than all the agents in the system suddenly being forced to act in whatever manner they can afford to act with 15 milliseconds of computation to decide.

[robbiep]

That’s also why there are trading halts built into all major exchanges, so if something triggers a large fall, there is a circuit breaker in the decision process of 18-48 hours for people to process and digest information

[gricardo99]

Just FYI, NYSE Rule 80B[1] halts trading for 15 minutes, or until the next day if the first 2 levels are breached. CME follows similar rules.

1 - http://wallstreet.cch.com/nyse/rules/nyse-rules/chp_1_3/chp_...

[greenyouse]

Just for fun here's a link to a 2010 flash crash before those rules were in place where a trillion dollars were dropped in half an hour due to algorithms.

https://en.wikipedia.org/wiki/2010_Flash_Crash

[scotty79]

Maybe if economy can be crashed in 15 seconds it should be crashed in 15 seconds do we can have a saner economy?

Letting AI in is like fuzzing for computer programs. It can break your program in interesting ways but that's a good thing.

[bartread]

> Maybe if economy can be crashed in 15 seconds it should be crashed in 15 seconds do we can have a saner economy?

How do you know that would be the outcome?

[adrianN]

Numbers getting smaller on the stock market doesn't destroy any real value. Companies still have all the IP they had before, all the real estate, all machines... If the bits in some computer's RAM are so disconnected from reality that they can change so dramatically without any catastrophe in the real world, then the rational choice is to disregard those bits.

[phildawes]

Crashed markets reflect a real difference in what people are prepared to pay for assets. That has a real direct impact on how people plan and behave.

[raz32dust]

Slower only because humans need more time to arrive at the same conclusions, and use more data points. Either one can be made true of computers too.

[tlb]

Qualitatively, humans are likely to say, "Whoa! Hold on. This doesn't make sense." when presented with unreasonable data, while algorithms are likely to proceed and do whatever.

We don't have good programming abstractions for "This doesn't make sense." You want something between proceeding and aborting.

[raz32dust]

Why not? Probability + threshold/activation function in neural nets is basically modeling that.

[new4thaccount]

I read once that a soldier in Russia was supposed to fire off nuclear warheads once when faulty sensors erroneously indicated that America had already done so.

Had the system been a mathematical contract, we would all be dead. Instead, a hero determined sat pondering whether he should do it and then realized the sensors had to have been defective as he wasn't dead yet.

[Zanni]

You're probably thinking of Stanislav Petrov: https://en.wikipedia.org/wiki/Stanislav_Petrov

[new4thaccount]

Yes and thanks for linking. I re-read it and according to the article he didn't actually have the power to press the button, but not immediately reporting to his superiors who had the "button" had the same impact and my previous statement remains true about automated systems built by imperfect beings.

[yters]

Humans don't think kittens are avocados with complete certainty because someone flicked a speck of green paint on the kitten.

[Viliam1234]

Before stating this with complete certainty, I would ask Oliver Sacks first.

https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_f...

[yters]

His patients realized something was wrong with their perception, unlike a computer.

[inimino]

Some of them did!

[raz32dust]

https://en.wikipedia.org/wiki/Proof_by_example

[exelius]

The biggest problem with “human intelligence” is that our memory is so short and influenced by emotion. Though judging by the amount of data created and lost over the last decade, that’s probably true of computers too.

[raz32dust]

But is emotion bad? Emotions can be thought of as programmed biases, and we might have evolved them because they are collectively good for the species. Emotion is what caused many of the examples stated in this thread where a human made a better decision than a computer.

[exelius]

I don't think emotion on its own is bad or good; but it's not something that a computer can emulate. You may get better decisions in some scenarios with certain stakeholders, and worse decisions in other scenarios.

When we talk about "programmed bias", we need to be careful because different subsets of the population hold different bias -- so the bias programmed in is reflective of the experiences of the team that developed the software rather than the population of users it affects.

[onemoresoop]

In the same vein, a machine should always be geared towards being a tool put into the use of the human intelligence. This tool was designed by human intelligence after all. The problem is when human intelligence creates such a tool that becomes so complex it no longer understands how it works, what it does and how to stop it.

There's no doubt that more things will be managed by machines and artificial intelligence but let's not make human intelligence a second class citizen.

[maxxxxx]

I think what will happen is that a few people at the top will manage the machines and benefit from them while they will be trying to make the other humans second class. I think a scenario like “Elysium” is very possible. That is until there is a revolution and then nobody knows how that will end.

[dwaltrip]

Without some robust general intelligence in the loop, whether that be AGI or human, computers will always have countless strange, unpredictable, and devastating failure modes, especially if given increasing responsibility.

This is the reason why few systems of significant complexity and risk exist today without humans in the loop.

[inflatableDodo]

>computers will always have countless strange, unpredictable, and devastating failure modes, especially if given increasing responsibility.

Sounds like most people I know. We are much better at reflecting on them though.

[inflatableDodo]

>I don't think it takes a math genius to see how this is a bad idea.

I agree, it has long been a key trope of sci-fi;

>We've crashed her gates disguised as an audit and three subpoenas, but her defenses are specifically geared to cope with that kind of official intrusion. Her most sophisticated ice is structured to fend off warrants, writs, subpoenas. When we breached the first gate, the bulk of her data vanished behind core-command ice, these walls we see as leagues of corridor, mazes of shadow. Five separate landlines spurted May Day signals to law firms, but the virus had already taken over the parameter ice. The glitch systems gobble the distress calls as our mimetic subprograms scan anything that hasn't been blanked by core command.

>The Russian program lifts a Tokyo number from the unscreened data, choosing it for frequency of calls, average length of calls, the speed with which Chrome returned those calls.

>"Okay," says Bobby, "we're an incoming scrambler call from a pal of hers in Japan. That should help."

from Burning Chrome, by William Gibson.

You could even argue that it is one of science fiction's founding tropes, as it can be traced back to the stories regarding golem.

[0815test]

Different roles. Computational contracts are not about e.g. the "traders" on a security exchange, they're about the exchange institution itself, and the precise definition of how it should work and interact with agents. Behind any sort of "smart market" there's a computational contract of some kind, and the converse may also be true - inasmuch as it interacts with multiple agents who jointly determine the 'prices' involved, a "computational contract" is a smart market, and its usefulness can fairly be assessed as such.