The exact same could be said about Freedom of Speech itself. Or any other fundamental freedom/privilege for that matter. Potential for abuse is grounds for mitigation, not wholesale curtailment.
No because Freedom of Speech is a fundamental individual right. It should never be limited or restricted. A strong country will prevail over abusers in this case (think when America abolished slavery, segregation...). It is a compass for long term survival.