[nopinsight]

Superintelligence (along with some definitions): https://en.wikipedia.org/wiki/Superintelligence

Also, "Dario Amodei says what he has seen inside Anthropic in the past few months leads him to believe that in the next 2 or 3 years we will see AI systems that are better than almost all humans at almost all tasks"

https://x.com/tsarnick/status/1881794265648615886

[hatefulmoron]

Not saying you're necessarily wrong, but "Anthropic CEO says that the work going on in Anthropic is super good and will produce fantastic results in 2 or 3 years" it not necessarily telling of anything.

[nopinsight]

Dario said in mid-2023 that his timeline for achieving "generally well-educated humans" was 2-3 years. o1 and Sonnet 3.5 (new) have already fulfilled that requirement in terms of Q&A, ahead of his earlier timeline.

[hatefulmoron]

I'm curious about that. Those models are definitely more knowledgeable than a well educated human, but so is Google search, and has been for a long time. But are they as intelligent as a well educated human? I feel like there's a huge qualitative difference. I trust the intelligence of those models much less than an educated human.

[nopinsight]

If we talk about a median well-educated human, o1 likely passes the bar. Quite a few tests of reasoning suggests that’s the case. An example:

“Preprint out today that tests o1-preview's medical reasoning experiments against a baseline of 100s of clinicians.

In this case the title says it all:

Superhuman performance of a large language model on the reasoning tasks of a physician

Link: https://arxiv.org/abs/2412.10849”. — Adam Rodman, a co-author of the paper https://x.com/AdamRodmanMD/status/186902305691786464

—-

Have you tried using o1 with a variety of problems?

[hatefulmoron]

The paper you linked claims on page 10 that machines have been performing comparably on the task since 2012, so I'm not sure exactly what the paper is supposed to show in this context.

Am I to conclude that we've had a comparably intelligent machine since 2012?

Given the similar performance between GPT4 and O1 on this task, I wonder if GPT3.5 is significantly better than a human, too.

Sorry if my thoughts are a bit scattered, but it feels like that benchmark shows how good statistical methods are in general, not that LLMs are better reasoners.

You've probably read and understood more than me, so I'm happy for you to clarify.

[nopinsight]

Figure 1 shows a significant improvement of o1-preview over earlier models.

Perhaps it’s better that you ask a statistician you trust.

[emaro]

Can they do rule 110? If not, I don't think they're 'generally intelligent'.

[philipwhiuk]

But there's 0 guarantee they are even capable of solving the rather large amount that covers the rest of a well-educated human.

[whiplash451]

Anthropic has to say this or Anthropic does not see their next funding round.