Sailors have figured this out centuries ago to travel against the wind (called tacking). Some of the same principles apply, like orienting the sail so that photons push against the sail reducing the angular momentum.
Tacking works because you have resistance against two media (air and water) which are travelling at different velocities -- you need a keel in the water. Solar sails don't have an analogous second medium.
"Sailing upwind" here -- if you were genuinely tacking, rather than doing something analogous to tacking -- would (with a big enough solar sail) let you propel yourself into the sun more quickly than gravity alone would pull you in.
So yes, you can produce a sideways component and let gravity do the rest, but that component can never actually push you towards the sun.
Sure, the analogy is loose, but as a sailor myself, I think it fits the fundamental ethos- one works the natural forces against each other to get where they are trying to go- wind, tides, and both aerodynamic and hydrodynamic lift.
The fact that one has a number of force vectors in varying directions with both sailing and solar sailing, both let you work them together in unison get where you want to go.
Quite often when I am "sailing" the wind is light or dies, and I go to "windward" by timing the currents, and wind at best lets me position for the right currents, but below a sufficient level produces no lift to windward.