That's completely outdated thinking if you reach the level of AI being described.
Complex systems could be completely self healing and self quarantining, "red team" can be freely interrogated "blue team" and convinced to create attacks that are then mitigated.
And again, the AI itself would improve at self-interrogation, so we're saying "trivialize", but trivial as in tricking a system capable of hacking into power grids with ease into ignoring its training.
People who go to this doomsday scenario fail to extend any sort of lateral thinking.
An LLM that trivializes taking down the power grid would not be "GPT 4 + SCADA infiltration", it'd be a new paradigm in how humanity operates.
Blue team has to imagine every possible exploit before it occurs for a system that is essentially black box, has unknown emergent behaviors and the input to the system is anything that can be described by human language.
Complex systems could be completely self healing and self quarantining, "red team" can be freely interrogated "blue team" and convinced to create attacks that are then mitigated.
And again, the AI itself would improve at self-interrogation, so we're saying "trivialize", but trivial as in tricking a system capable of hacking into power grids with ease into ignoring its training.
People who go to this doomsday scenario fail to extend any sort of lateral thinking.
An LLM that trivializes taking down the power grid would not be "GPT 4 + SCADA infiltration", it'd be a new paradigm in how humanity operates.