What difference does it make with planes ? I understand your fears and fears in general but a slow, low noise, low energy (actually I should check that) thing like a balloon is charming to me.
It has been some time since reading it, but hot air balloons are much more prone to accidents than airplanes. So much can go wrong where modern aircraft are designed to lessen these sorts of things. We just sensationalize plane crashes - because they are horror-inducing spectacles - and don't really hear about the balloon incidences that hurt 3 people instead of 200.