A lot of regret actually comes from enjoying something and then later realising that it was stupid and a waste of time. Time that you would have liked to spend on a more useful endeavour.
A lot of your personal regret might come from that. I think it's important to note that there are many people who can look back fondly at times they were simply enjoying themselves, even if that time was not strictly 'useful'.