[robbomacrae]

I'm generally a proponent of AI and LLM but to me the decision was the right one. You are tasking people with implementing an idea generated by an algorithmic model with (I'm guessing) zero oversight that might have very little training that teaches it the importance of coming up with ideas worth implementing. Some may be more useful than others so it won't be fair from an accomplishment or motivation point of view.

Imagine you've already invested time going to this event and want to win the prize/credit but to do so you have to implement a plugin that makes webpages grayscale because of a random idea generator. Maybe some people would find that interesting but others would see it as wasting their time.

[jimmySixDOF]

Individual ideas can be subject to Ethical Review Board approvals and that should go for a hackathon project same as any study proposed in Academia or drug trial etc -- but to apply some wavey handed lum sum out of bounds lable just based on source seems like arbitrary opinionated overreach.

[golol]

As long as all participants are well-informed then there is absolutely no ethical issue...

[rsfern]

How do you make sure the participants are well informed? What if an idea suggested by a model turns out to be dangerous to implement, but nobody at the hackathon has quite the relevant experience to notice?

[naasking]

Such as?

[HeatrayEnjoyer]

rsfern is asking exactly that

[naasking]

No, I'm asking for an example of an idea that an LLM might produce that is too dangerous to implement but nobody at the hackathon has the relevant experience to notice. You can shut down any endeavour by imagining boogeymen that aren't actually real.

[rsfern]

I don’t think anything needs to be shut down necessarily, I’m just suggesting reasons why a reasonable ethics board might be hesitant to green light such a hackathon if it’s not clear the organizers have done their due diligence on safety

I might be biased in terms of the safety profile, my background is materials and chemistry, and there are loads of ways you can get into trouble if you don’t really have experience in the materials and synthesis routes you’re working with

One example I’ve heard of from my field (alloy design) is an ML model that suggested an composition high in Magnesium - perfectly reasonable if you’re interested in lightweight strong alloys, but the synthesis method was arc melting, which is a high risk for starting a metal fire if you aren’t careful because Mg has a low vapor pressure

If you’re doing organic chemistry it’s maybe even worse because there can be all kinds of side products or runaway exothermic reactions, and if you’re doing novel chemistry it might take deep experience in the field to know of those things are likely

All these concerns are manageable, but I think an ethics review panel would want to at least see that there is a reasonable safety review process in place before letting students try out random experiments in topic areas in which the models likely haven’t been fine tuned with safety in mind.

[taneq]

Surely the ideas themselves are what should be examined for ethical suitability, rather than the meta-idea of “ask an LLM for ideas”?