The pointer and the accelerometers are two separate things. I haven't heard many complaints about the pointer accuracy (which is what Rayman uses for the section you're thinking of).
The Wii sensor bar combined with the little camera inside your Wiimote can give you a reasonably accurate estimate of where your wiimote is pointed on the screen.
This is quite bad at gestures though, particularly when the front of the Wiimote is pointed away from the sensor bar (as it is in games like boxing and tennis). In these cases, lacking a visual way to tell movement, the controller relies entirely on the accelerometer, which while theoretically capable, saturate very quickly with full body motion.
This is why in Raving Rabbids, where you have to play rabbid-DDR with the music, you have to jerk the controller - the accelerometer is neither reliable nor accurate enough to reliably infer your intentions through light motions.