This is a really interesting question. Is it? My intuition would say no since you have no inherent duty to protect or help others. I have no clue though.
If there's no downside to leaving the ladder in place, then I would think yes - there's a reasonable expectation that people will die due to your actions. You'd likely have to argue about "involuntary manslaughter" vs something more intentional though, depending on circumstances.
If there is a reasonable downside, probably no? You have a right to try to keep yourself alive, in nearly all contexts.