You could instead speculate that what we consider the "Visual Areas" of the brain are more than that. Hence we have been wrong about those parts of the brain.