For early societies, you might be thinking of the Greeks, though apparently it's somewhat common between languages?[1] For Greek the idea to originate from Homer's writings, with colors being more described in terms of how light or dark they are versus specific hues.[2]