Most of them don't have a history of direct hostility towards the US and specifically having the willingness to allow a US foe to use their proximity to plant tactical weapons to threaten the US.
Castro was already playing footsie with the Soviets and Castro nationalizing US business interests was one of the reasons we decided to invade.
That was 64 years ago and are you suggesting here that things that happened that far back have no bearing today? Hell, the brother of the dictator from 64 years ago is currently in charge there today still oppressing the Cuban people so it’s not like there hasn't been some continuity.