It’s not impossible to switch majors at CP SLO. My wife was a physics major at CalPoly SLO until she got tired of being the only girl in all her classes and switched to biochem.
Sorry to hear that. It was a big issue when I was there (mid 2000's). Not particularly friendly folks either. I hung out with music majors as much as I could