I let mine lapse a couple years ago because I realized I don't actually care that much if a package comes in 2 days or 3-5. I then noticed a lot of my Amazon shipments were coming in 3 days anyways.
I've found that in recent years, non-Prime orders will sit at the "ordered" phase for an extra few days, before finally being shipped at the same speed as a prime order.
I thought it might've been a holiday crush sort of situation, but it continued well past the new year (2019)
I'm almost certain that at the scale Amazon works at, there has to be a priority queue for these sorts of things. There's no way they could make Prime work if they just changed the shipping method to make it faster, there has to be order fulfilment prioritization too.