While it depends on your perspective, I'll contend the Swiss referendum system has also disadvantaged certain groups of people as well. One example of that was the 2009 minaret referendum:
It's an example of using a referendum system to religiously disenfranchise a minority population. Further, much of the imagery used to persuade the population to vote for the referendum, such as the Muslim woman wearing a chador across the backdrop of a Swiss flag covered by minarets that look like missiles, was bigoted and defamatory.
My point in raising this incident is to provide an example that a direct referendum system has advantages and disadvantages. One of the disadvantages seen here is that the system can be used to remove a right from minority population important to their cultural identity. More broadly, there's always a tension between majority rule and protection of minority rights in a democracy and both the California and Swiss referendum systems allow those lines to be redrawn by populist appeal.