Yes, but if they enforce that, they might fire you but they won't be able to take your IP away. Not if they exclude the IP you produce for other companies as part of the IP agreement.
When you sign up to work for the company, you agree to do X and they agree to pay you Y. You are disloyal to them if you don't do all or part of X (for instance, if X includes thinking up ideas for them, taking your really valuable ideas that you thought up while working for them and leaving and using them yourself), and they are disloyal if they don't pay you Y. Those are the terms, after all! So I don't see how loyalty only goes one way...