Due to noise and aliasing. If multiple features (edges, materials etc) cover a single pixel, you usually get aliasing. If you do random sampling (of reflections, lights, whatever) you get noise.
For anti-aliasing you'll usually want at least on the order of 10 rays per pixel for a nice result. If you do random sampling, you quickly need 100 to 1000 rays per pixel to get an acceptable noise level.
One ray represents many photons, and those photons scatter off in many different directions when they hit most surfaces. To get a reasonable picture of where they went, you need a lot of samples.
A single raycast also only gets you one step of the light's journey. That means, for example, that following light as it bounces from a light source, to a surface, to another surface to your eye requires at least 3 rays even for perfectly mirrored surfaces.
It depends on what you wan't to use ray tracing for.
If you want to calculate global illumination with path tracing (which is a form of ray tracing) then you need multiple samples per pixel to get a noiseless result.
It's a stochastic process and it needs many samples per bounce. Basically, unless the surfaces are perfect mirrors, each angle of reflection is a probability function. Look for Monte Carlo methods for more information.
For the secondary rays for reflection, refraction and shadows.
At one ray per pixel, you can only do the primary ray and find which triangle it hits. The exact same thing can be done with rasterization much faster and with equal results.
For anti-aliasing you'll usually want at least on the order of 10 rays per pixel for a nice result. If you do random sampling, you quickly need 100 to 1000 rays per pixel to get an acceptable noise level.