It's even a problem with allies as well as adversaries; countries can potentially get around their own legal limitations by agreeing to spy on one another, and where needed to build a case, covering their tracks via parallel construction. Whether officially or unofficially, any backdoor keys would leak almost immediately.