Nonexistence -- both in the sense of me not being here anymore, being forgotten, and everything connected to or similar to me disaapearing -- bothers me way more than dying. I'm also not too fond of the idea that the universe might wind down to nothing, with no future or memory of its past.
This used to bother me. I used to fear that all that is human would ultimately be lost and unknown to a cold and dispassionate universe.
It doesn't bother me any more, as my premise was wrong. I came to realise that the only way the universe can know us is through us - and that the best record of any moment is that perception of it lent there and then to those who lived through it - but ultimately the retrospective reminiscence of a gone past is but a pale shadow of the real, the here and now.
Cherish life, and strive for the future for its own sake - but never forget that what matters is here and now.
Come to think of it, I think what most fear is not death, but having never lived.
It's just a larger scale of leaving high-school (or leaving any significant period in your life behind). Things, people, problems from back then don't bother you anymore and you remember them only a little bit better than a novel you read, being out of context. Also you dive in nonexistence everyday when in deep sleep. And the universe winding down, well, like you said, its just an idea :-)