Size and number of lanes, for me. I think roundabouts are the multi-lane ones, rotaries are much smaller and slower usually. At least this is what I realized on moving from Massachusetts where rotaries were common, to New Jersey where there were many "roundabouts" which required lane changes and no reduction of speed.
rotary: tangential entry/exit, large diameter, high-speed, multi-lane
roundabout: deflected entry/exit, small diameter, low-speed, sometimes single-lane