Technically the formula for force is F = dp/dt, or the derivative of momentum with respect to time.
For particles with mass the momentum of such a particle is p = mv, and so you can use that to yield F = ma. However for a massless particle like a photon, its momentum is p = E / c. If you use that momentum to describe a beam of photons being absorbed by a material, then you get F = n * E / c, where E is the average energy of the photons, n is the rate of photons per second, and c is the speed of light.
Things that move at the speed of light carry momentum, even though they have zero mass. For something moving at the speed of light, the momentum is the energy divided by c. (You can't have something with mass moving at the speed of light.)
Since the photon has nonzero energy E and is moving at c, it has mass by virtue of E=mc^2. And since E=hv, the E part is determined by the photon's frequency v, so the equivalent amount of mass is as well. It shows up as radiation pressure when the photon hits an object, just as if something tangible had collided with it.
On a sunny day, I'd guess that sunlight exerts about the same force on an acre on the Earth's surface as a postage stamp lying on the ground.
E=mc^2 doesn't mean that a photon has mass. Photons do produce gravitational effects due to their energy, but not in the same way as a particle with mass m. See, for example, https://physics.stackexchange.com/a/6222
Objects in motion have kinetic energy, and Einstein says mass and energy are equivalent. This means in a very real sense objects in motion have additional "relativistic mass". When you annihilate that photon it's energy is transferred to whatever absorbed it.
Confused? You're not alone! Physicists are trying to move away from the terms "rest mass" and "relativistic mass" for reasons including one you've already identified: what does it even mean for a photon to be "at rest"?
For particles with mass the momentum of such a particle is p = mv, and so you can use that to yield F = ma. However for a massless particle like a photon, its momentum is p = E / c. If you use that momentum to describe a beam of photons being absorbed by a material, then you get F = n * E / c, where E is the average energy of the photons, n is the rate of photons per second, and c is the speed of light.