Lower quality towels have a very loose weave that easily comes apart both in the washing and in the drying. Some of the very worst, in my case Eastern European, towels are so bad that they show near bald spots after just 3 washing cycles.
I can't see any problems with this; those are tiny fibers and they don't make a big difference except over years. It doesn't affect the weave that much.
(My towels are a very distinct shade of red, a color I never wear. The lint is the exact same color.)