[kcorbitt]

In 1850, the year California became a state, its population made up 0.4% of the country.

Today, Californians are 12% of the US population. It's just an accident of history that all those people happen to count as one "state" – if it were to join the union today as a collection of states of the same relative size as 1850 California, that would be 30 states. If it were to split into individual states each with the population of today's Wyoming, that would be 67 states.

So yes, the unit of federalism is the state. But the way state lines are drawn are largely an accident of history, so state-based representation feels pretty arbitrary.