For an organization requiring the highest available security, the ideal solution would be a privately operated hardware security module kept off the DMZ. However, that, as well as the idea of self hosting (and maintaining) the entire dev, test, deploy, and prod stack suggested by another commenter, isn't always within reach of a small, agile team looking to focus on their core competencies.
One could argue that it's possible for Amazon to have falsified the description of KMS as an HSM, or the certifications[0] they were granted for it, but I'd retort that an organization in a position to seriously question those claims shouldn't be using a remote solution anyway.
So, making the more rational assumption that such claims by Amazon can be trusted, their offering is quite secure: the HSM does not allow the export of any key, and exposes only the ability to load encrypted data into the device and have it produce the decrypted result over a secure channel, and vice versa.
I said it above, but I'll reiterate here that Amazon KMS does not use HSMs; they don't provide a lot of detail to help you reason about what that implies for key security. (I agree that there's no reason to believe they're lying or that it's backdoored.) There's also not much discussion about where the authorization checks happen, and the security of key operations is only as secure as the entity to whom that is delegated.
Looking at the docs, it looks like the master key source is pluggable, so you don't have to use Amazon's KMS... but none of the other options inspire confidence (local file, fetch from URL, plaintext password, or no password).
At the very least, I'd like to see a plugin for using a key stored on a local TPM chip -- which almost any modern bare-metal server would be equipped with.
For an organization requiring the highest available security, the ideal solution would be a privately operated hardware security module kept off the DMZ. However, that, as well as the idea of self hosting (and maintaining) the entire dev, test, deploy, and prod stack suggested by another commenter, isn't always within reach of a small, agile team looking to focus on their core competencies.
One could argue that it's possible for Amazon to have falsified the description of KMS as an HSM, or the certifications[0] they were granted for it, but I'd retort that an organization in a position to seriously question those claims shouldn't be using a remote solution anyway.
So, making the more rational assumption that such claims by Amazon can be trusted, their offering is quite secure: the HSM does not allow the export of any key, and exposes only the ability to load encrypted data into the device and have it produce the decrypted result over a secure channel, and vice versa.
[0]: https://aws.amazon.com/kms/details/#compliance