With experience and some anxiety resistance filler words can be filtered out as you said. It does detract ones attention from the speaker and reduces the quality of the talk. It is a long process.
It's well worth the effort as it it will also pay dividends in the longer term. Speaking wo filler words in normal conversations (especially in the workplace) can help you immensely.