[quarterwave]

Very good question. Here's the gist of it, a rough way of thinking about it.

A time varying electromagnetic field is produced by moving charges (except for a minor distinction about steady currents). The direction of motion of the charges determines the polarization of the emitted field. Later on, when the field encounters matter it produces an identical motion of charges i.e; if originally an East-West charge motion produced a North-South field then a N-S field will induce an E-W motion among new charges which in turn re-emit N-S light. The original light and the re-emitted light interfere, the delay is what slows the light down, as noted by others. But you can see why the direction is the same.

Note: I suppose this picture is somehow tied to reciprocity of the Maxwell equations between sources and fields, I need to go look it up again.