Yes. My attitude is that I've entered into a commercial exchange with my employer. They pay me money and in exchange get a set amount of my undivided time and attention.
They have no legitimate claim over what I do during the time that I'm not being paid. I may contractually agree to certain narrow restrictions (such as not talking about the company or the details of my work), but generally, my time is my time to spend as I see fit. In my worldview, any other stance is my employer making some sort of ownership claim over my life. That's unacceptable to me.
If what I do in my off time adversely impacts my performance on the job, then my employer can reprimand or fire me for poor performance.
Well, true. However, assuming good intentions, there are laws against working two/three jobs in the world in a manner that would rob you of your time to rest, which was presumably put in place to prevent a lot of undesirable things from happening. So far, I had no problems with side jobs as long as I agreed with my employer and made sure that my work/rest times are not in danger (nobody ever said no).