Both referendums were widely boycotted for various reasons by no voters. The only people who voted were the small minority who wanted to become a state.
The 2012 election had 78% turnout with valid ballots in the question of whether or not to continue the current status, on which a majority voted to reject the current status, and 56% turnout on the question of alternative status, on which a majority chose Statehood.
On either question, the turnout was better than several US Presidential elections (and on the first better than any recent Presidential election.)
There were significant boycotts of the 2017 referendum because of the wording of the title, but in any other US election, we don't ignore election results because people choose to abstain.
It looks like approximately 1.8 million people voted on the first question, and 1.36 million people voted on the second. And in 2016 Puerto Rico was estimated as having a population of 3.4 million. In light of that, 1.8 million votes on the first and 1.36 million votes on the second seems fairly similar to the normal voter turnout in the U.S.
So no, it doesn't sound to me like a "small minority" of people were voting.
EDIT: This is talking about the 2012 referendum.
EDIT 2: It occurs to me that 3.4 million isn't the voting population, it's the total population. So the voter turnout is even better than it looks initially.
A small minority of Americans voted for Trump. Should we have ignored the results of the election because only a small minority of sour grapes shouldn't make that important a decision for the rest of us?
On either question, the turnout was better than several US Presidential elections (and on the first better than any recent Presidential election.)
There were significant boycotts of the 2017 referendum because of the wording of the title, but in any other US election, we don't ignore election results because people choose to abstain.