[KhoomeiK]

A group of PhD students at Stanford recently wanted to take AI/ML research ideas generated by LLMs like this and have teams of engineers execute on them at a hackathon. We were getting things prepared at AGI House SF to host the hackathon with them when we learned that the study did not pass ethical review.

I think automating science is an important research direction nonetheless.

[srcreigh]

That’s pretty wild. What was the reason behind failing ethics review?

[robbomacrae]

I'm generally a proponent of AI and LLM but to me the decision was the right one. You are tasking people with implementing an idea generated by an algorithmic model with (I'm guessing) zero oversight that might have very little training that teaches it the importance of coming up with ideas worth implementing. Some may be more useful than others so it won't be fair from an accomplishment or motivation point of view.

Imagine you've already invested time going to this event and want to win the prize/credit but to do so you have to implement a plugin that makes webpages grayscale because of a random idea generator. Maybe some people would find that interesting but others would see it as wasting their time.

[jimmySixDOF]

Individual ideas can be subject to Ethical Review Board approvals and that should go for a hackathon project same as any study proposed in Academia or drug trial etc -- but to apply some wavey handed lum sum out of bounds lable just based on source seems like arbitrary opinionated overreach.

[golol]

As long as all participants are well-informed then there is absolutely no ethical issue...

[rsfern]

How do you make sure the participants are well informed? What if an idea suggested by a model turns out to be dangerous to implement, but nobody at the hackathon has quite the relevant experience to notice?

[naasking]

Such as?

[HeatrayEnjoyer]

rsfern is asking exactly that

[taneq]

Surely the ideas themselves are what should be examined for ethical suitability, rather than the meta-idea of “ask an LLM for ideas”?

[CJefferson]

One obvious problem is, what if the ideas were obviously unethical?

I would personally let this pass ethics if someone read all the generated ideas, and took personal responsibility for them passing the basic ethics rules, or got them through the ethics committee if required, exactly the same as they would their own ideas.

[brigadier132]

I don't think LLMs are the right approach for this. Coordinated science would basically be a search problem where we verify different facts using experiments and use what we learn to determine what experiment to do next.

[visarga]

When you can run experiments quickly it becomes feasible to use ML and evolutionary methods to do novel discoveries, like AlphaTensor's better matrix multiplication than Strassen, and AlphaZero's move 37, upturning centuries of game strategy.

The paper "Evolution through Large Models" shows the way. Just use LLMs as genetic mutation operators. Evolutionary methods are great at search, LLMs are great at intuition but get stuck on their own, they combine well. https://arxiv.org/abs/2206.08896

The interplay between LLMs and Evolutionary Algorithms, despite differing in objectives and methodologies, share a common pursuit of applicability in complex problems. Meanwhile, EA can provide an optimization framework for LLM's further enhancement under black box settings, empowering LLM with flexible global search capacities.

Since chatGPT was first released hundreds of millions of people have been using it for assistance, and the model outputs influenced their actions, maybe even supported scientists to make new discoveries. The LLM text is filtered through people and ends up as real world consequences and discoveries that are reported in text, and get in the next training set closing the loop.

Trillions of AI tokens per month do this slow feedback game. AI speeds up the circulation of useful information and ideas in human society, and AI feedback gets filtered by the contact with people and the real world.