They store energy but there's a limit to how quickly you can get that energy out, hence a limit to power. And I suppose you can kinda talk about them "storing power" in the sense that power is what you might be interested in getting out of them.
Power is just the time derivative of energy, like velocity is the time derivative of position. Both batteries and capacitors store energy, which they can release at some given peak and average power. Batteries tend to focus more on energy density (energy stored per weight), while capacitors tend to focus more on power density (rate of energy stored or released per weight).
I think that just reflects that the typical usage for a capacitor is smoothing out some very spiky load that runs at a high frequency, say a microcontroller running at 100MHz. The average power needs of the MCU are low, but it's drawing almost all of it in a big burst on every clock tick (10ns).
Power supplies handle that badly, and the pulses turn your supply traces in a big antenna and suddenly you're an unlicensed FM radio station.
So for the capacitor that's put in place to buffer that load, the total capacity is important, but what really matters is that the part can manage that 100MHz charge/discharge cycle.
At least, that's my understanding of it.