The more complicated answer is that empty space is expanding, but matter is not. Gravity, and matter subjected to the various forces, holds together even as empty space expands around it.