[nathan_compton]

I'm a physicist and I have published research in neuroscience. I don't feel like you understand the material you are talking about. Let's consider the question of formal verification. It is entirely possible for a computer to generate a proof for any proposition for which a person can generate a proof if by no other means than randomly printing out characters that represent the proof and then checking if the proof is correct. A proof is always a finite number of characters and the verification of a proof is always a finite number of steps (otherwise we wouldn't be able to write it down and to convince ourselves that the proof is true).

There are certainly problems which are hard for computers which are easy for people (and vice versa) but I don't know of any formal version of the claim that people can perform actions which are in principal impossible for human beings. You seem to be alluding to the claims of Penrose that humans can somehow do things machines cannot, but this isn't even what Penrose is claiming in the book. Instead, Penrose claims humans _somehow perceive_ the truth of propositions which cannot be proven in a given formal system, but this is a very vague claim which few scholars take seriously. He also thinks quantum mechanics is involved somehow, but this is also entirely vague. Even if one were to demonstrate that quantum mechanics were somehow involved in cognition there isn't any account that I know of that would describe how, per se, merely having a quantum computer involved would somehow introduce computations that classical computers could not do. I mean for one thing one can always simulate any quantum process on a classical computer. I'm not an expert in complexity theory, but as far as I know there are no known problems for which it is proven that there are no non-deterministic classical algorithms that perform as well as quantum algorithms. So we cannot even demonstrate with certainty that quantum computers are in fact better than classical computers with randomness. We only have situations where quantum algorithms are theoretically better than any known classical algorithm. It is also worth noting that most neuroscientists are dubious about the importance of quantum mechanical effects in cognition.

All that said, some people believe that there is some property of people that allows them to do things that computers cannot do (at the very least generate or have consciousness). But I don't know of any way of supporting that idea that doesn't involve dualism of some kind. That is, that asserts that there is something more than material that somehow constitutes a person. Like I said, a lot of people believe that there is. Most people. But a plurality of scientists do not believe this. I do not believe it. While I wouldn't say I'm a materialist, I would say I am a monist and that belief basically makes the assertion that a machine can do what a person does trivially true.

All that aside, I agree with you that language models cannot reason, almost certainly have no consciousness, and are have the propensity to be bad for people "spiritually" (speaking loosely of course). I'm not sure things are quite as dire as you assert. The moral rot of working in a call center, indeed, the existence of call centers, is surely worse than the imposition of an AI agent between the caller and the worker.

[trod123]

Apologies for not being able to respond sooner. I don't know what is wrong with YN, but when attempting to respond a number of times now over the past few days I've repeatedly received the YN error message saying you've posted too much in too short a time, despite only having posted what's shown up on my public dashboard (which is hardly anything).

As for formal verification of software, there are often common problems with decidability that go ignored, or design problems within the model being tested where similar looking but ultimately different objects are treated the same in error (apples to oranges). This latter part is often quite common with abstraction and its acting as a conceptual black box.

Unfortunately, this is not just a simple brute force problem. Required properties for the system need to be preserved, and these properties may or may not be explicitly defined in the formal model under test, not to mention that many tools designed for this purpose also have a corresponding combinatorial explosion with regards to complexity in many cases. Termination as a property is one that is quite common, and famous in Computer Science (i.e. the Turing Halting problem).

We can design a model that will guarantee termination, but we cannot easily determine if an arbitrary model will terminate without first running it. The hard part is in the definition and coming up with the proof of correctness, not necessarily the actual verification of correctness part.

The former as far as I'm aware is always done by humans because machines above a trivial complexity will get stuck in loops largely due to the halting problem, or the predictability paradox (which also has roots in halting). They will chug away without realizing they've repeated the same steps ad infinitum. Inductive proofs as a class are one that a lot of machines choke on or run forever.

It seems to me like you may be confounding my meaning of 'non-determinism' in a context from Floyd, instead of from a system's property context (such as found in EE coursework on Signals and Systems).

The former context is a bit more relaxed than the latter (i.e. its similar to the same way as describing the absence of time in-variance [system property]), while the latter requires unique inputs to unique output maps (i.e. preserving that 1:1 function test at each step of a computation) The property basically acts as a virtual rail for the CPU states to follow some computational graph with at most only one edge being followed based on inputs.

Von Neumann Architectures are all inherently deterministic at the signal domain level, the instructions are static and usually in the same place, and its processed in the CPU (also a single fixed place, at a single time). The preserved property guarantees that each unique instruction will will always have a single unique output (given care with the design of the instructions), and will provide that same output regardless of time, absent further compromises as you go up the abstraction layers of the architecture.

Non-determinism can be introduced through coding of interfaces and other methods of abstraction that may break the property (such as flattening, shifting, etc).

Overall non-determinism's usefulness is vanishingly small (mostly being bugs, errors, and undefined behavior), and its introduction breaks any guarantees that further automation (which may be passed output from non-deterministic code) may work at all.

Non-determinism is largely uncharacterizable (outputs don't corresponding to input as potential causes) and as a result lacks the properties for troubleshooting (aside from a guess and check model), and in practice exception handling cannot generally correct these type of errors given the unpredictability.

Troubleshooting of digital systems is largely just an application of the structured properties. Absent those properties, you cannot isolate the problem without full knowledge of the system including its design spec for inputs and outputs (which often isn't available, or economic).

Unfortunately, the role of determinism or its absence as a system's properties doesn't allow any direct comparison without knowledge of the architecture. While we can make some assumptions, we wouldn't have any sound basis because the devil really is in the details with regards to how superposition can be leveraged, and the instruction set.

Sorry if this seemed a bit run-on. I find it is important to be clear without ambiguity with the meaning of the words which is why I'm clarifying (also given that I may not be able to respond timely if YN keeps blocking me). Hopefully this was sufficiently clear in identifying any discrepancies of underlying meaning.

> I'm not sure things are quite as dire as you assert.

The direness of the assertion comes from long experience and intimate understanding of feedback systems, and a realization of the dependencies for acting on problems.

Generally speaking, Feedback systems are normally considered broken when they no longer appropriately react to signals they should normally react to.

Perception in people is just one such feedback system, and there are structures of stimuli that can bypass it in most people. Cialdini has documented a number of these in his book Influence. Other areas such as Reflected Appraisal and its distortion, along with destructive interference have also been investigated related to our biological stages of development for self concept the main way we transfer culture to our offspring (Ericksonian Identity), as well as some subtle applications of Sapir-Whorf in the 1950s on PoWs (which follow Cialdini's Consistency Principle).

The general rule of thumb for feedback systems are when they continue running, once broken the longer they take to catastrophic breakdown the higher the overall cost.

In an ecological overshoot environment, where failure of society to organize (such as happens in a breakdown) or even react leads to a failure to feed ourselves, it requires a critical view based solely on the existential risk.

Now if your perception is changed without you recognizing or alerting to it, such that you no longer recognize certain problems and discount them from your awareness, you become blind, and without knowing/being able to react you cannot act to correct it. Critically thinking people may be able to cope better than others, but no one I know is hyper-rational all the time.

Perception underlies most of what we do, so when these structures and the following consequences happen, you ultimately get stuck in a feedback loop as an unthinking automata without realizing it. You may perform regular fixed action patterns but its still the same output oblivious to the danger signals you may be receiving.

Associative priming of unrelated things is one such method discovered in the past 70 years, similar to attenuating signals (jamming), or front of line blocking (more common in bureacracy); those are the two most common ways these systems fail.

Would you not consider a subtle, indirect, and blinding madness that is uncharacterizable but spreading and overtaking people, and chaotically increasing across a population a greater imposition than an AI agent crudely disrupting people's ability to communicate (which the underlying mechanic this agent performs has been shown to break people; with a cohort trending towards this type of psychotic behavior or alternatively dissassociating completely as automata)?

Its been well documented that the first thing to go in torture is often rational thought, which would be our sole defense.

Authors Meerloo and Lifton both cover case studies of these type of mental torture structures which have been used both by Nazi's and Mao, as well as so haphazardly and carelessly today by businesses towards manipulating sentiment and marketing, none more commonly than social media and adtech.

Meerloo covers WW2 Nazi Thought Reform through Mao (where they broke perception). Lifton documents in detail the case studies from victim POWs in the Korean Conflict (Mao). These structures are even commonly found in schools.

Personally, I think a blinding illness that spreads subtly through communication, that you can't react to without first being inoculated against is a pretty dire threat.

Predictability Paradox and links to Halting (overview) https://link.springer.com/content/pdf/10.1007/s10670-020-003...

[nathan_compton]

All very interesting, but much of it is besides the simple point I am trying to make. I know of no formal demonstration that definitively distinguishes between the kinds of processes a person can do and the kinds of processes a computer can do in principal. In practice various processes are easy for computers and hard for people and vice versa, but any product which manifests itself as a finite document (clearly all proofs are finite documents) whose validity can be determined in a finite number of steps can obviously be created by a computer although getting it to do so may be hard. Indeed, the production of proofs is quite hard for human beings as well.

One other note:

> Overall non-determinism's usefulness is vanishingly small .

This is really not true. There are many problems which have no useful deterministic solution but for which non-deterministic solutions exist which can get very close to an optimal answer, and non-determinism is extremely important. In fact, in complexity theory Turing machines with and without a random oracle are distinguished precisely because the former is so useful and, in many respects, the big question in the complexity theory of quantum computers comes down to: is the randomness of a quantum computer more powerful than mere classical randomness?

People perform certain kinds of proof tasks in practice better than computers, in my opinion, only because we have a large volume of heuristic strategies for proving things which are embedded in the culture of mathematics and which are difficult but in principal not impossible to embed in a machine. In a sense, humans prove things by _guessing correctly_ which turns out to be how we get computers to do these kinds of non-trivial search tasks as well.

Anyway, on the cultural end of things I agree with the basic idea that some applications of language models and other AI are probably bad for society, but I think you're just overstating the case that this particular technology is so catastrophic. In fact, this technology merely takes effort out of a person and puts it into a machine: in a call center situation if a person gets angry at the call center employee, that is itself an error on the part of the caller, since the person at the center isn't personally responsible for anything and is, in fact, placed there almost entirely to sop up the emotional energy of the customer. The call center employee in turn has to do the internal work of "throwing away" the emotions of the customer so as to continue to respond in the robotic way that their bosses require them to perform. Merely being in a call center is intensely dehumanizing and, presently, places the burden of that dehumanization on the employee: they must dehumanize _themselves_ or face the economic consequences. This technology displaces some of that labor. Does it make the entire "call center" locus more dehumanizing? Hard to say, in total, but if so, its marginal. The employee in this situation is by far the one with the shortest end of the stick and I have trouble getting really bent out of shape about a technology which might make their lives slightly more pleasant.

If the product in question here were a secret AI agent that intermediated between a partners in a romantic relationship or between parent and child (especially) then I would get your point, but the call center is such a moral morass that this piece of technology hardly matters.

Speaking more broadly: you have a grand narrative, clearly, in your mind, but I think its preventing you from looking closely at the things that are really happening and responding rationally to them. In particular, in the cases to which you refer, where various sorts of strategies to disrupt cognition were so damaging, one thing stands out in particular: the overwhelming, credible, threat of actual violence to the subject from a difficult or impossible to escape force. Without that critical ingredient I would expect most of the effects you describe to be missing or highly attenuated.