I'm surprised to hear a native Californian would say "back east." I thought only people from the East Coast (or I guessed who lived there at one point) would go "back."
West coast Canada we also "back east" -- it's the linguistic opposite of "out west".
The underlying source is the idea everyone is somehow "from" the east, and they (or their ancestors) at one point moved "out west", but if they were to return, it would be to go "back east."
I'm in California now (and have been for over half my life, at this point), but I suspect I'd say "back east" when talking about my past, at the very least. For me, there's a sense of time involved, as well as direction.
As a California native, I say "back east" with the same connotation as a farmer saying "the back 40" -- somewhere annoying to get to and somewhat overgrown.
The underlying source is the idea everyone is somehow "from" the east, and they (or their ancestors) at one point moved "out west", but if they were to return, it would be to go "back east."