I don't think it's really to do with accent, but rather that non-native speakers tend to talk slower and use simpler grammar and a more limited vocabulary (including fewer idioms).
That does help, I’m sure. There’s also a tendency to incorrectly apply foreign grammatical structures to the language in a way that might be strange or incomprehensible to a native speaker while making perfect sense to a fellow second-language speaker.