Well, our brains are only wired up to detect a limited spectrum of colors. There are insects and birds that can detect a greater variety of color. So, how can it just exist in our brains?
One way to look at colors is to see them as qualia, which is basically a subjective experience. Electromagnetic waves come in a large range of frequencies, some of which excite the nerve endings in your retina, causing your brain to experience colors. But colors are a bit more complicated than that: for example, there is no such thing as just "yellow": yellow is the subjective term given by humans when there is a certain balance between the excitation of their there-color vision. You can experience the exact same sense of "yellow" from different types of light: there is true monochromatic yellow around 580nm, but you can also experience that same yellow when mixing the right amount of red and green light. There is no wave of 580nm in the second case, but still you see the same color.
Our eyes are wired up to detect a limited spectrum of wavelengths. Although those wavelengths correspond to a color, not all colors correspond to a wavelength. That’s why OP stated that color is a brain phenomenon.