I like that analogy. An even better one: porting the code of `sed' to a new operating system without fully understanding it, using `sed' to automatically correct bugs in fundamental operating system code without checks, recompiling said OS, and hoping it works perfectly.
> It's like editing a piece of code without fully understanding what it does, without unit tests, and your text editor may be buggy.
No, it is far worse than what you described. your described buggy code can cause damages but you have the choice to delete them all to stop any further damages.
There is no such option for experiments done on humans.
> There is no such option for experiments done on humans.
There are, unfortunately, options for deleting humans with "buggy code", the real problems surround the fact that it is other - probably "buggy" - humans making the determination of which humans to mark for deletion.
And on top of that, the piece of code is fully spaghetti, poorly understood and documented, and contains tight coupling between components. You don't know what breaks and the implication, until it really breaks.