[naasking]

No, I'm asking for an example of an idea that an LLM might produce that is too dangerous to implement but nobody at the hackathon has the relevant experience to notice. You can shut down any endeavour by imagining boogeymen that aren't actually real.

[rsfern]

I don’t think anything needs to be shut down necessarily, I’m just suggesting reasons why a reasonable ethics board might be hesitant to green light such a hackathon if it’s not clear the organizers have done their due diligence on safety

I might be biased in terms of the safety profile, my background is materials and chemistry, and there are loads of ways you can get into trouble if you don’t really have experience in the materials and synthesis routes you’re working with

One example I’ve heard of from my field (alloy design) is an ML model that suggested an composition high in Magnesium - perfectly reasonable if you’re interested in lightweight strong alloys, but the synthesis method was arc melting, which is a high risk for starting a metal fire if you aren’t careful because Mg has a low vapor pressure

If you’re doing organic chemistry it’s maybe even worse because there can be all kinds of side products or runaway exothermic reactions, and if you’re doing novel chemistry it might take deep experience in the field to know of those things are likely

All these concerns are manageable, but I think an ethics review panel would want to at least see that there is a reasonable safety review process in place before letting students try out random experiments in topic areas in which the models likely haven’t been fine tuned with safety in mind.