Almost all of the founding fathers either owned slaves or weren't willing to stick their necks out to stop it. We absolutely should question their judgement.
Politics is the art of the possible. Given what post-slavery America looked like for the former slaves (not much different), the question becomes: do you care so much about slavery, which is an ancient institution, that you give up the idea of forming a united front?
Was it good? No, it was a bad idea. But that's something we say comfortably from our homes in a large, powerful country. In 1787... this was a remote and weak place. If you wanted the slave-dependent colonies to join, you had to either buy their slaves from them or allow it to continue. And there wasn't enough money to buy them.
$googlesearch "how many signers of the US Constitution were slave owners?"
25 Of the 55 delegates to the Constitutional Convention, about 25 owned slaves. Many of the framers harbored moral qualms about slavery.
Historical Context: The Constitution and Slavery