Why not? Why does your presence imply your responsibility? In the given trolly problem, we can infer how events would unfold in your absence. Why does your presence imply they should unfold any differently? Shouldn't responsibility be voluntarily accepted, rather than imposed by default?
How can you impose a duty to fight evil when "evil" itself has no workable concrete definition?
Assuming the full question is:
> Is it not possible to be a passive observer of evil without being responsible to some degree?
No, with the caveats of being mentally and physically able to do something about it of course.