Yup, I have this as well. 1 is black, 2 is yellow, 3 is green, 4 is orange, and so on. It has a side-effect: it's easier to remember phone numbers because I visualize sequences of colors (239 is yellow-green-brown, etc).
I can understand something like a phone number, where each digit is singular and stands apart from the others. If you're saying that about composite numbers (like 'two hundred and thirty nine'), you're the first person I've met claiming that word/number-color synesthesia helps them remember composites/sums.