Simple example: say I use S3 with the Amazon-managed key. Anyone who has a policy granting access to a bucket — and lots of people love to write bucket policies with Resource:* — can read or write objects in that bucket using that key.
If you use a CMK, you can write custom policies which are both far more restrictive and shouldn't change as much as other resource types. That means that I could, say, have a policy which says the key-admin role/group/user is the only principal in the account which can update the key settings at all (even administrator can't do that then), the writer role is the only one with kms:Encrypt, and the reader role is the only one with kms:Decrypt. No matter what S3 access you have, if you aren't one of those roles you won't be able to use the encrypted data. This is probably also in a scenario something like “central group A provisions the KMS, devops group B creates lots of other resources using that key”.
You can add conditions, too — “anyone in our account can encrypt, decrypt requests can only come from this IP address or VPC”, “only requests from this AWS service are accepted” (i.e. that compromised EC2 instance can't use it), “access to data encrypted with this key can only be done in the two regions we approve of that”, “this particular encryption value must be used on all requests”, etc.
That adds an extra layer of defense: if I compromise a user, even one with some level of administrative access, who doesn't have the CMK access all I can get are errors or encrypted data rather than the raw data. If you're careful you can architect environments where a person can deploy code without direct access to secrets or a system can stream data through to an encrypted store (if you are storing PII, this can be a huge difference between “all of our users” and “only the ones who used the system during this time period”). In some cases these can be bypassed (i.e. a CMK might not allow Administrator to access it directly but they could possibly issue credentials for a user who does have access) but you're preventing generic attacks which just scrape up everything that compromised credentials have access to, and hopefully increasing both the level access required and the likelihood of producing an audit alert.
If you use a CMK, you can write custom policies which are both far more restrictive and shouldn't change as much as other resource types. That means that I could, say, have a policy which says the key-admin role/group/user is the only principal in the account which can update the key settings at all (even administrator can't do that then), the writer role is the only one with kms:Encrypt, and the reader role is the only one with kms:Decrypt. No matter what S3 access you have, if you aren't one of those roles you won't be able to use the encrypted data. This is probably also in a scenario something like “central group A provisions the KMS, devops group B creates lots of other resources using that key”.
You can add conditions, too — “anyone in our account can encrypt, decrypt requests can only come from this IP address or VPC”, “only requests from this AWS service are accepted” (i.e. that compromised EC2 instance can't use it), “access to data encrypted with this key can only be done in the two regions we approve of that”, “this particular encryption value must be used on all requests”, etc.
https://docs.aws.amazon.com/kms/latest/developerguide/key-po...
https://docs.aws.amazon.com/kms/latest/developerguide/policy...
That adds an extra layer of defense: if I compromise a user, even one with some level of administrative access, who doesn't have the CMK access all I can get are errors or encrypted data rather than the raw data. If you're careful you can architect environments where a person can deploy code without direct access to secrets or a system can stream data through to an encrypted store (if you are storing PII, this can be a huge difference between “all of our users” and “only the ones who used the system during this time period”). In some cases these can be bypassed (i.e. a CMK might not allow Administrator to access it directly but they could possibly issue credentials for a user who does have access) but you're preventing generic attacks which just scrape up everything that compromised credentials have access to, and hopefully increasing both the level access required and the likelihood of producing an audit alert.