The direction is "towards" or "away" from the sun, the only way to get closer to the sun is to make your orbit smaller, the only way to do that is to slow down in your current orbit. You can't reach a lower orbit by accelerating. (I'm simplifying a bit, it's all acceleration, what matters is if it's in your orbital velocity, prograde, or against it, retrograde)
You can accelerate so slow down. The flyby of a planet creates an accelleration that, if in the correct direction, results in a slower orbital velocity in the sun's reference frame. In really simple terms, if you approach the planet from the right (in a counterclockwise solar system) you speed up, if you aim to the left you slow down. The velocity is transferred to/from the planet.
Notice how the probes orbit gets more eccentric but still smaller each time it approaches a planet on the left. The gravity assist is slowing it down, again in the solar reference frame.
Gravitational slingshots are a method were you accelerate and end up slowing down but I don't think it's fair to say that you "just accelerate to reach a lower orbit", since your acceleration ultimately ends up being either retrograde for very low fly-by orbits or on the normal/radial axis. Or possibly even prograde.
Basically, using a flyby to apply retrograde speed is not much more than accelerating prograde and then using the gravity of a planet to A) move your orbital position without a speed change and/or B) redirect your acceleration.
It's all relative. You can use gravity assists to get closer to the sun, but there are a lot more planets above our orbit than below it (and mercury is too small to be useful). ...so timing gravity assisted deceleration orbits is not just hard, but it takes forever and the only reasonable solution is a very eccentric orbit.