They are invasive (in the US) and grow like weeds completely taking over and crowding out native species. Having many of them screws up the nitrogen level of the soil further crowding out/killing native plants.
Other commenters have covered most of the reasons why they are awful, but my main complaint is that they will completely take over your garden beds and wooded areas if left undisturbed.
This alters the ecosystem.