We decided to decrypt secrets on service startup so if KMS is down during the deploy we can stop the deploy after first server fails to start. Not perfect but good enough for our use case.
In case of catastrophic KMS failure we can always manually replace secrets with plaintext and revoke them afterwards.
In case of catastrophic KMS failure we can always manually replace secrets with plaintext and revoke them afterwards.