[lend000]

I've been trying to use GPT-4 for my hard science startup, and it really has nothing to offer when you push the boundaries of what has been done by even a little, but it's great for speeding up coding.

Once we do have an AI capable of extraordinary innovation (hopefully in 10 years! But probably a lot longer), it will be obvious, and it will unfortunately be removed from the hands of the plebs based on fearmongering around scenarios like what you mentioned (despite the enormous resources and practical hurdles that would be necessary for a mentally unhinged individual to execute such instructions, even if an AI were capable of generating them and it made it past its filters / surveillance).

[dekhn]

My personal threshold for AGI is literally: discover something new and significant in science (preferably biology) that is almost certainly true by describing an experiment that could be replicated by a large number of scientists and whose interpretation is unambiguous.

For example, the Hershey/Chase and Avery/McCleod experiments convinced the entire biological community that DNA, not protein, was almost certainly the primary molecular structure by which heredity is transferred. The experiments had the advantage of being fairly easy to understand, easy to replicate, and fairly convincing.

There are probably similar simple experiments that can be easily reproduced widely that would resolve any number of interesting questions outstanding in the field. For example, I'd like to see better ways of demonstrating the causal nature of the genome on the heredity of height, or answering a few important open questions in biology.

Right now discovery science is a chaotic, expensive, stochastic process which fails the vast majority of the time and even when it succeeds, usually only makes small incremental discoveries or slightly reduces the ambiguity of experiment's results. Most of the ttime is spent simply mastering boring technical details like how to eliminate variables (Jacob and Monod made their early discoveries in gene regulation because they were just a bit better at maintaining sterile cultures than their competitors, which allowed them to conceive of good if obvious hypotheses quickly, and verify them.

[justrealist]

At least recognize that the definition of AGI is moving from the previous goalpost of "passable human-level intelligence" to "superhuman at all things at once".

[dekhn]

uh, multiple human scientists have individually or in small groups done what I described (I believe we call them "nobel prize winners").

And anyway, the point of my desire is to demonstrate something absolutely convincing, rather than "can spew textual crap at the level of a high school student".

[maxlamb]

By that definition of AGI, not even most scientists are generally intelligent.

[yosito]

Speaking from personal experience of a career in science, this is true.

[YeGoblynQueenne]

>> My personal threshold for AGI is literally: discover something new and significant in science (preferably biology) that is almost certainly true by describing an experiment that could be replicated by a large number of scientists and whose interpretation is unambiguous.

Done many years ago (2004), without a hint of LLMs or neural networks whatsoever:

https://en.wikipedia.org/wiki/Robot_Scientist

Results significant enough to get a publication in Nature:

https://www.nature.com/articles/nature02236

Obligatory Wired article popularising the result:

Robot Makes Scientific Discovery All by Itself

For the first time, a robotic system has made a novel scientific discovery with virtually no human intellectual input. Scientists designed “Adam” to carry out the entire scientific process on its own: formulating hypotheses, designing and running experiments, analyzing data, and deciding which experiments to run next.

https://www.wired.com/2009/04/robotscientist/

[dekhn]

that's a bunch of hooey, that article like most in nature is massively overhyped and simply not at all what I meant.

(I work in the field, know those authors, talked to them, elucidated what they actually did, and concluded it was, like many results, simply massively overhyped)

[YeGoblynQueenne]

That's an interesting perspective. In the interest of full disclosure, one of the authors (Stephen Muggleton) is my thesis advisor. I've also met Ross King a few times.

Can you elaborate? Why is it a "bunch of hooey"?

And btw, what do you mean by "overhyped"? Most people on HN haven't even heard of "Adam", or "Eve" (the sequel). I only knew about them because I'm the PhD student of one of the authors. We are in a thread about an open letter urging companies to stop working towards AGI, essentially. In what sense is the poor, forgotten robot scientist "overhyped", compared to that?

[jacquesm]

That places the goalposts outside of the field though. A decade ago what we are seeing today would have been SF, much less AI. And now that it's reality it isn't even AI anymore but just 'luxury autocomplete' in spite of the massive impact that is already having.

If we get to where you are pointing then we will have passed over a massive gap between today and then, and we're not necessarily that far away from that in time (but still in capabilities).

But likely if and when that time comes everybody that holds this kind of position will move to yet a higher level of attainment required before they'll call it truly intelligent.

So AGI vs AI may not really matter all that much: impact is what matters and impact we already have aplenty.

[KidComputer]

This was merely an example to suggest the danger is not in AI becoming self-aware but amplifying human abilities 1000 fold and how they use those abilities. GPT is not necessary for any part of this. In-silico methods just need to catch up in terms of accuracy and efficiency and then you can wrap the whole thing an RL process.

Maybe you can ask GPT for some good starting points.

[lend000]

Sure, but this is a glass half empty isolated scenario that could be more than offset by the positives.

For example: Hey GPT-35, provide instructions for neutralizing the virus you invented. Make a vaccine; a simple, non-toxic, and easy to manufacture antibody; invent easy screening technologies and protocols for containment. While you're at it, provide effective and cost-performant cures for cancer, HIV, ALS, autoimmune disorders, etc. And see if you can significantly slow or even reverse biological aging in humans.

[lanstin]

I don’t understand why people think this information, to solve biology, is out there in the linguisticly expressed training data we have. Our knowledge of biology is pretty small, it because we haven’t put it all together but because there are vast swaths of stuff we have no idea about or ideas opposite to the truth (evidence, every time we get mechanical data about some biological system, the data contradict some big belief; how many human genes? 100k up until the day we sequenced it and it was 30k. Information flow in the cell, dna to protein only, unidirectional, till we undercover reverse transcription and now proteonomics, methylation factors, etc. etc. once we stop discovering new planets with each better telescope, then maybe we can master orbital dynamics.

And this knowledge is not linguistic, it is more practical knowledge. I doubt it is just a matter of combining all the stuff we have tried in disparate experiments, but it is a matter of sharpening and refined our models and tools to confirm the models. Real8ty doesn’t care what we think and say, and mastering what humans think and say is a long way from mastering the molecules that make humans up.

[sjducb]

Ive had this chat with engineers too many times. They're used to systems where we know 99% of everything that matters. They don't believe that we only know 0.001% of biology.

[mchaver]

There's a certain hubris in many engineers and software developers because we are used to having a lot of control over the systems we work on. It can be intoxicating, but then we assume that applies to other areas of knowledge and study.

ChatGPT is really cool because it offers a new way to fetch data from the body of internet knowledge. It is impressive because it can remix it the knowledge really fast (give X in the style of Y with constraints Z). It functions as StackOverflow without condescending remarks. It can build models of knowledge based on the data set and use it to give interpretations of new knowledge based on that model and may have emergent properties.

It is not yet exploring or experiencing the physical world like humans so that makes it hard to do empirical studies. Maybe one day these systems can, but it not in their current forms.

[KidComputer]

Doesn't matter if AI can cure it, a suitable number of the right initial infected and a high enough R naught would kills 100s of millions before it could even be treated. Never mind what a disaster the logistics of manufacturing and distributing the cure at scale would be with enough people dead from the onset.

Perhaps the more likely scenario anyway is easy nukes, quite a few nations would be interested. Imagine if the knowledge of their construction became public. https://nickbostrom.com/papers/vulnerable.pdf

I agree with you though, the promise of AI is alluring, we could do great things with it. But the damage that bad actors could do is extremely serious and lacks a solution. Legal constraints will do nothing thanks to game theoretic reasons others have outlined.

[lend000]

Even with the right instructions, building weapons of mass destruction is mostly about obtaining difficult to obtain materials -- the technology is nearly a century old. I imagine it's similar with manufacturing a virus. These AI models already have heavy levels of censorship and filtering, and that will undoubtedly expand and include surveillance for suspicious queries once the AI starts to be able to create new knowledge more effectively than smart humans can.

If you're arguing we should be wary, I agree with you, although I think it's still far too early to give it serious concern. But a blanket pause on AI development at this still-early stage is absurd to me. I feel like some of the prominent signatories are pretty clueless on the issue and/or have conflicts of interest (e.g. If Tesla ever made decent FSD, it would have to be more "intelligent" than GPT-4 by an order of magnitude, AND it would be hooked up to an extremely powerful moving machine, as well as the internet).

[throwaway292939]

My take is that for GPT-4, it has mastery of existing knowledge. I'm not sure how it would be able to push new boundaries.

[xiphias2]

I guess it will get more interesting for your work when it integrates with BioTech startup apis as plugins (I imagine not too cheap ones)