If your hand is on the track switch, you're just as responsible for the trolley no matter which way it goes. Walking away from the switch does not absolve you.