He's not popular with some people for what might be an abuse of power.[1] He's had a reputation of intervening in architecture projects that don't fit his ideal vision (ie. they're not traditional, Georgian style), leading to them being cancelled outright even at the late stages of planning.
See also Edward VII -- http://en.wikipedia.org/wiki/Edward_vii -- Queen Victoria's oldest surviving male offspring, he had to wait 59 years for the throne. Reigned for nine years, then died.
Given the Windsor family medical history, Elizabeth II could well make it past her centenary. In which Charles will not get to place his bum on the throne until he's in his late seventies or eighties.
Quite a lot of people have mentioned (not particularly seriously) that they would prefer if he just abdicated for William (who is very popular).