There's nothing wrong with having a referendum over it. But a horrible media industry has made a circus over the entire thing, leaving many people misinformed.
But that is exactly what is wrong with having a referendum over it. The circus is inevitable, as are emotional arguments abused by populist politicians. Some matters shouldn't be decided by which side of an argument happens to have hired the better team of PR consultants.