Obviously he means the American culture and by our he means Americans. There's no denying that if immigration is high enough, and assimilation poor enough, culture will change very quickly. Not saying its the case in the US but it's simple to understand what he meant.
You say that, but there are plenty of people with books and spots on TV that say the exact same things about 'culture' and they mean 'white' (and say it explicitly). It's a way for people to make an argument about something that doesn't need numbers or data, since immigrants rapidly become a net positive on the economy and commit less crime.