No, I don’t think people think that… because that isn’t what it means. People aren’t annoyed or sarcastic towards you… they are just pausing for a second.
Another name for ellipses (…) is “suspension point” and doesn’t have any true grammatical purposes, unlike a comma. A comma doesn’t necessarily indicate a pause and makes no sense to insert randomly… wherever you’d like the reader to pause.