If they didn't vote because they assumed some result, perhaps they are not as passionate about their position and should not get to vote a second time.
Quite a lot of the not voting were young people who had not figured the logistics of registering a month before and so on. The oldies who votted for brexit had mostly voted before and so were pre registered and know the procedure which slewed the results.
The same thing could be said to people that vote but don't understand what they are voting about. A lot of of the Exit votes were a way to express disappointed to the state of things of their lives.