You could also see it the other way around, your application is not secure enough if terrorists, dissidents, drug dealers, whistleblowers and child pornographers don't feel confident to use it.
You could but I doubt the general public and policy makers are going to see it this way. They're still looking for a backdoor that only the "good guys" can access.
From an ethical point of view building this software must be difficult. On one hand you are building something that advances technology and could be used to help free people from an oppressive government, on the other hand you are also building something that could be used (and if it works most likely will be used) to aid acts that we all agree are morally wrong.