A human driver would also follow the same heuristic. That is why it was programmed that way. People don't assume people are going to randomly walk in front of their cars except at cross walks.
Idk about you, but if I see a human start to step into the road I slow down/stop depending on how close they are. Someone stopped and facing the road looking both ways obviously may be attempting to cross, even if "illegally"
I'm enough of a jerk that sometimes, if they are at the edge of the road, I will stop and honk. Even if they are engrossed in their phone. Because, in my jurisdiction, it is illegal to not give the pedestrian the right of way at a crosswalk. Pedestrians who aren't planning on entering the roadway shouldn't stand at a crosswalk and not enter.