It can be hard to avoid though especially in the US. I was amazed at the difficulty there of just getting a loaf of bread without sugar mixed in. In Europe they don't usually have sugar.
The sugar in American bread is ostensibly to allow yeast to ferment quicker than they would on flour alone, to impart bread-y flavors, not for sweetness. This shortens processing time.