[unkulunkulu]

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.)

This coupled with people acting on its predictions is a kind of self fulfilling prophecy.

which is to ask, are AI safety folks building models of this pattern? :)

[godelski]

This is true for a lot of things, not just AI. But in AI, a guy who didn't get a High School degree and wrote Harry Potter Fan Fiction is one of the leading voices in doomerism.

The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts.

What's funny to me is that people will confidently argue with experts and others value their opinion over the expert's knowledge. But on the other hand, people tend to just take machines at face value. Maybe these aren't overlapping groups, but it does appear that way. There's a great irony in trusting a machine but not the person/s that built said machine.

[elicksaur]

I don’t trust the machines or the people who make them, and I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments. What group does that make me?

[godelski]

> I don’t trust the machines or the people who make them

This makes you consistent. I have no problem with this.

> I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments

I mostly agree. But my point is that Eliezer Yudkowsky doesn't actually have the qualifications. I want to be clear that academic degrees aren't necessary to qualify someone, just like they aren't necessary to qualify someone as a good programmer. But it is generally harder and the foundation is shakier. In this case, most of his arguments are founded on incorrect assumptions. They are often logical, but it doesn't matter if something is logical if the premise is incorrect.

[FinchNova12]

Not sure if you intended this, but it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents. Maybe you are distinguishing between doomers vs broader AI Safety proponents, but if not, I feel like the counterargument is that most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk) interface quite frequently with these AI models and are also (purportedly) seriously concerned about AI safety

[godelski]

> it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents

It's a not so thinly veiled critique of Eliezer Yudkowsky.

> Maybe you are distinguishing between doomers vs broader AI Safety proponents,

I do. These are different classes of people. But many doomers mascaraed as AI Safety proponents. Just as many conmen mascarade as ML/AI researchers. I suspect distinguishing the groups is quite difficult for those without domain expertise.

> most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk)

I don't care about the opinion of most of these people (there are some I VERY much do), nor do I think this is a meaningful letter.

Interfacing with a model does not endow one with any level of expertise. If this were true, the whole thread would be ill founded because people using GPT are interfacing with it. Instead, one needs to actually deeply study these models. There are things we know about them, and quite a lot. The term "blackbox" gets thrown around a lot, but that doesn't make everyone's expertise on the matter equally valid. In fact, the more complex something is to understand suggests the fewer number of people are qualified to have a reasonable opinion on the matter. My complaint is we often act as if the opposite is true.[0]

My second big problem with the CAIS letter is it means nothing. All it says is "I don't want to kill all humans." This is a fairly universally agreed upon statement and is in fact the default statement. It does not say anything about the potential risk. That's a completely different matter.

Worse, many of the people who have signed this are literally at the helm of the ships steering us into a dystopian future (which is not covered by this toothless letter). So I'm not sure what meaning this is supposed to have other than pageantry. Do not forget that these are the same exact people pushing and promoting abuse of these tools. I do not blame Average Joe for thinking that GPT is equivalent to Google (which itself cannot be trusted at face value, but this does not make it a useless tool) when that is often the way that it is promoted/advertised. So if you are concerned, I wouldn't use this as evidence.

[0] There's an added problem that you can become above average in any given subject relatively quickly. This is a double edged sword because knowledge is valuable but it often results in one being over confident. And the learning difficulty grows exponentially, which is why there are so few experts in any given subject matter. Because expertise is understanding nuance and complexity. The great irony of the doomers is that they fall back on "unknown unknowns" while not putting effort towards putting a bound on that.

[SrslyJosh]

> are AI safety folks building models of this pattern?

First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.

[immibis]

We have already created Skynet. Its name is Capitalism. Or the Internet. One of those things.

[bamboozled]

I know your downvoted but I don’t think you would be if you had of just said, corporations.

I think oil companies are the greatest existential threat to humanity via lobbying and eventually climate destruction. Second is social media companies. It’s so easy to spread misinformation against a whole populace and it’s going south fast.

There is just too much incentive for those vested in these corporations to just stop.

[throwanem]

How would that work, do you think?

[wegfawefgawefg]

If you knew everyone would ask gpt before doing anything, you would make gpt say what woudl generally be considered the better option. Not going to war, not committing suicide, etc. In this way even if war was the optimal decision according to some other utility function, the behavior of people is directed in a positive way. (Presumably)

[throwanem]

Sure, if you also assume people follow whatever advice so given. They won't, even before the covert influence effort becomes popular knowledge, as it inevitably will. This destroys consumer trust in your product after you have successfully made that product indispensable, thus opening up a previously impossible vacuum in epistemology and thus access to power.

[wegfawefgawefg]

For the record I believe it to be immoral to manipulate humanity in this way. And I also believe it might be bad for bussiness.

I was just trying to explain to the guy above what I think the guy above that meant.

[throwanem]

'The guy above' is also me :) And yeah, I get it. I guess the thing I'm trying to get at by extension and example is just how hard a problem this is, and maybe also that the formulation given assumes LLMs have a level of control over human behavior that not only doesn't exist but in the general case sort of can't since the LLM's user is always free not to take its advice. (At the very least, if humans have no option but to follow LLM instructions, something has gone much more badly wrong than the risk of there being poor instructions...)

In general, I think it's a good example of the kind of social problem tech can make a lot worse but no better: when a society has lost its grasp of epistemology, multiplying the amount of information available, at a net decrease in quality and reliability, merely multiplies the scope of the problem.

[gerdesj]

"accuracy2" sigh - the 2 is a superscript to a footnote and not a domain specific term.

"facially impossible" ... does that really riff on "on the face of it", or is it farcically misspelt?

Garbage in, garbage out 8)

[tikhonj]

"Facially" in the sense of "on the face of it", roughly as a synonym for "obviously", seems like a pretty standard usage to me—this is certainly not the first place I've seen the word used in this sense.

[riverdweller]

Human recall failure. Probably wanted "seemingly", "apparently", or even "ostensibly", but who's got time for all that when the publish button's right there.

[fragmede]

Also from the article:

> Also, ML code tends to vastly more complex and less standardized than traditional statistical modeling.

I mean, hey, it's proof that the text isn't AI generated, since ChatGPT is better at English than that, but it makes it hard to read and I'm not going to buy their book if it's going to be full of errors like that.