[kirkules]

The existence of "The dress", which is seen by different people or the same person in different states of mind as white/gold or blue/black, and other such illusions, would seem to invalidate your perspective.

Or if that's not persuasive, just consider different software versions, different video drivers, etc. running on the same hardware.

Or consider that if you believe a tumor in your visual cortex could meaningfully change color perception in your model, then why wouldn't a difference in cortex structure have some effect too? I'm willing to bet no two humans have ever had or probably will ever have the same arrangement of visual cortex neurons and other relevant brain biology. Do you think you know enough to say any such variation is irrelevant?