Then leave the less important thing if you want to do the more important thing.
After all, nothing is stopping you from doing the politics outside of work.