Friday, December 14, 2018

AWS Secrets Manager Cross account for SES credentials

If you have external systems sending emails, it might only know of stmp with username and password.

When you want to get CIS compliance in AWS having an access keys last longer than 90 days takes you out of compliance, as such you need to change this access key every 3 months. This seems to be a bit of pain.

Luckily AWS released Secrets Manager to allow as you guessed it secrets, to be rotated.

This is an extension on https://aws.amazon.com/blogs/security/how-to-access-secrets-across-aws-accounts-by-attaching-resource-based-policies/ since blog post is a bit too cryptic for actual implementation with least privileges.

To do that you firstly need to create a KMS key to use since the defaults don't allow addition policies to be added.

For extra security, the SES email sending service is centralised but the application needing to use the credentials are in another part of your AWS Organisation accounts. This is good security practice as you can then remove access to a kms key if a downstream account was compromised.


Here is an example policy which would allow any account in your Organisation to access the key as well as the role which the lambda function will use to rotate the secret
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-4",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${AWS::AccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${AWS::AccountId}:role/${your admin role you use}"
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion",
"kms:TagResource",
"kms:UntagResource"
],
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::${AWS::AccountId}:role/KeyRotationLambdaRole",
"arn:aws:iam::${AWS::AccountId}:role/${your admin role you use}"
]
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${AWS::AccountId}:role/${your admin role you use}"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
},
{
"Sid": "Allow use of the key cross account for smtp secrets manager",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GetKeyPolicy"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "${YOUR-ORG-ID from organisations}"
}
}
},
{
"Sid": "Allow access through AWS Secrets Manager for all principals in the account that are authorized to use AWS Secrets Manager",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "${AWS::AccountId}",
"kms:ViaService": "secretsmanager.${AWS::Region}.amazonaws.com"
}
}
},
{
"Sid": "Allow direct access to key metadata to the account",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${AWS::AccountId}:root"
},
"Action": [
"kms:Describe*",
"kms:Get*",
"kms:List*",
"kms:RevokeGrant"
],
"Resource": "*"
}
]
}

Please note: going with  "aws:PrincipalOrgID": "${YOUR-ORG-ID from organisations}" means that any account in your Org will have access to encrypt or decrypt with this key if they also have privileges to see this key (this means * * Administrators would by default have this, could could be changed to Principals with role names but then that next gotcha is that if the role is deleted and recreated, the connection to this kms policy would be disconnected)

Zip up this script and call it aws-key-rotation-lambda.zip
import boto3
import json
import logging
import os
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
"""Secrets Manager Rotation Template
This is a template for creating an AWS Secrets Manager rotation lambda
Args:
event (dict): Lambda dictionary of event parameters. These keys must include the following:
- SecretId: The secret ARN or identifier
- ClientRequestToken: The ClientRequestToken of the secret version
- Step: The rotation step (one of createSecret, setSecret, testSecret, or finishSecret)
context (LambdaContext): The Lambda runtime information
Raises:
ResourceNotFoundException: If the secret with the specified arn and stage does not exist
ValueError: If the secret is not properly configured for rotation
KeyError: If the event parameters do not contain the expected keys
"""
arn = event['SecretId']
token = event['ClientRequestToken']
step = event['Step']
# Setup the client
service_client = boto3.client('secretsmanager', endpoint_url=os.environ['SECRETS_MANAGER_ENDPOINT'])
# Make sure the version is staged correctly
metadata = service_client.describe_secret(SecretId=arn)
if not metadata['RotationEnabled']:
logger.error("Secret %s is not enabled for rotation" % arn)
raise ValueError("Secret %s is not enabled for rotation" % arn)
versions = metadata['VersionIdsToStages']
if token not in versions:
logger.error("Secret version %s has no stage for rotation of secret %s." % (token, arn))
raise ValueError("Secret version %s has no stage for rotation of secret %s." % (token, arn))
if "AWSCURRENT" in versions[token]:
logger.info("Secret version %s already set as AWSCURRENT for secret %s." % (token, arn))
return
elif "AWSPENDING" not in versions[token]:
logger.error("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn))
raise ValueError("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn))
if step == "createSecret":
create_secret(service_client, arn, token)
elif step == "setSecret":
set_secret(service_client, arn, token)
elif step == "testSecret":
test_secret(service_client, arn, token)
elif step == "finishSecret":
finish_secret(service_client, arn, token)
else:
raise ValueError("Invalid step parameter")
def create_secret(service_client, arn, token):
"""Create the secret
This method first checks for the existence of a secret for the passed in token. If one does not exist, it will generate a
new secret and put it with the passed in token.
Args:
service_client (client): The secrets manager service client
arn (string): The secret ARN or other identifier
token (string): The ClientRequestToken associated with the secret version
Raises:
ResourceNotFoundException: If the secret with the specified arn and stage does not exist
"""
# Make sure the current secret exists
current_dict = get_secret_dict(service_client, arn, "AWSCURRENT")
# Now try to get the secret version, if that fails, put a new secret
try:
get_secret_dict(service_client, arn, "AWSPENDING")
logger.info("createSecret: Successfully retrieved secret for %s." % arn)
except service_client.exceptions.ResourceNotFoundException:
iam_user = current_dict['smtpUser']
iam = boto3.client('iam')
# Remove the oldest key if there are multiple
logger.debug("Retrieving keys for user %s" % iam_user)
keys = iam.list_access_keys(UserName=iam_user)['AccessKeyMetadata']
logger.debug("Found %s key(s)" % len(keys))
if len(keys) > 1:
oldest_key = keys[0]
for key in keys:
if key['CreateDate'] < oldest_key['CreateDate']:
logger.debug("%s is older than %s" % (key, oldest_key))
oldest_key = key
key_id_to_delete = oldest_key['AccessKeyId']
key_date_to_delete = oldest_key['CreateDate']
iam.delete_access_key(UserName=iam_user, AccessKeyId=key_id_to_delete)
logger.info("Deleted old key %s from %s" % (key_id_to_delete, key_date_to_delete))
# Generate a new access key
logger.info("createSecret: Creating new access key for %s." % iam_user)
newKey = iam.create_access_key(UserName=iam_user)['AccessKey']
current_dict['smtpAccessKeyId'] = newKey['AccessKeyId']
current_dict['smtpSecretKey'] = newKey['SecretAccessKey']
current_dict['smtpUsername'] = newKey['AccessKeyId']
current_dict['smtpPassword'] = generate_ses_password(newKey['SecretAccessKey'])
# Put the secret
service_client.put_secret_value(SecretId=arn, ClientRequestToken=token, SecretString=json.dumps(current_dict), VersionStages=['AWSPENDING'])
logger.info("createSecret: Successfully put secret for ARN %s and version %s." % (arn, token))
def generate_ses_password(secret_key, charset='utf-8'):
import hmac #required to compute the HMAC key
import hashlib #required to create a SHA256 hash
import base64 #required to encode the computed key
# These variables are used when calculating the SMTP password. You shouldn't
# change them.
message = 'SendRawEmail'
version = '\x02'
# Compute an HMAC-SHA256 key from the AWS secret access key.
signatureInBytes = hmac.new(secret_key.encode(charset),message.encode(charset),hashlib.sha256).digest()
# Prepend the version number to the signature.
signatureAndVersion = version.encode(charset) + signatureInBytes
# Base64-encode the string that contains the version number and signature.
smtpPassword = base64.b64encode(signatureAndVersion)
# Decode the string and return it
return smtpPassword.decode(charset)
def set_secret(service_client, arn, token):
"""Set the secret
This method should set the AWSPENDING secret in the service that the secret belongs to. For example, if the secret is a database
credential, this method should take the value of the AWSPENDING secret and set the user's password to this value in the database.
Args:
service_client (client): The secrets manager service client
arn (string): The secret ARN or other identifier
token (string): The ClientRequestToken associated with the secret version
"""
# This is where the secret should be set in the service
# NB This is not applicable for access keys, since they are created and set in one operation
def test_secret(service_client, arn, token):
"""Test the secret
This method should validate that the AWSPENDING secret works in the service that the secret belongs to.
Args:
service_client (client): The secrets manager service client
arn (string): The secret ARN or other identifier
token (string): The ClientRequestToken associated with the secret version
"""
# This is where the secret should be tested against the service
pending_dict = get_secret_dict(service_client, arn, "AWSPENDING")
iam_user = pending_dict['smtpUser']
logger.info("Checking that the pending key is valid for user %s" % iam_user)
iam = boto3.client('iam')
logger.debug("Retrieving keys for user %s" % iam_user)
keys = iam.list_access_keys(UserName=iam_user)['AccessKeyMetadata']
logger.debug("Found %s key(s)" % len(keys))
pending_key = pending_dict['smtpAccessKeyId']
for key in keys:
if key["AccessKeyId"] == pending_key:
logger.info("Pending access key %s was found on the user" % pending_key)
return
raise ValueError("Unable to find pending access key on user %s" % iam_user)
def finish_secret(service_client, arn, token):
"""Finish the secret
This method finalizes the rotation process by marking the secret version passed in as the AWSCURRENT secret.
Args:
service_client (client): The secrets manager service client
arn (string): The secret ARN or other identifier
token (string): The ClientRequestToken associated with the secret version
Raises:
ResourceNotFoundException: If the secret with the specified arn does not exist
"""
# First describe the secret to get the current version
metadata = service_client.describe_secret(SecretId=arn)
current_version = None
for version in metadata["VersionIdsToStages"]:
if "AWSCURRENT" in metadata["VersionIdsToStages"][version]:
if version == token:
# The correct version is already marked as current, return
logger.info("finishSecret: Version %s already marked as AWSCURRENT for %s" % (version, arn))
publish_update(service_client, arn)
return
current_version = version
break
# Finalize by staging the secret version current
service_client.update_secret_version_stage(SecretId=arn, VersionStage="AWSCURRENT", MoveToVersionId=token, RemoveFromVersionId=current_version)
logger.info("finishSecret: Successfully set AWSCURRENT stage to version %s for secret %s." % (version, arn))
publish_update(service_client, arn)
def publish_update(service_client, arn):
logger.info("Notifying consumers that the secret has updated")
current_dict = get_secret_dict(service_client, arn, "AWSCURRENT")
topic_arn = current_dict['snsTopic']
if not topic_arn:
logger.warn("No topic ARN provided; unable to notify")
return
sns = boto3.client('sns')
result = sns.publish(TopicArn=topic_arn, Message='Updated %s' % arn)
logger.info("Notification sent as message %s" % result['MessageId'])
def get_secret_dict(service_client, arn, stage, token=None):
"""Gets the secret dictionary corresponding for the secret arn, stage, and token
This helper function gets credentials for the arn and stage passed in and returns the dictionary by parsing the JSON string
Args:
service_client (client): The secrets manager service client
arn (string): The secret ARN or other identifier
token (string): The ClientRequestToken associated with the secret version, or None if no validation is desired
stage (string): The stage identifying the secret version
Returns:
SecretDictionary: Secret dictionary
Raises:
ResourceNotFoundException: If the secret with the specified arn and stage does not exist
ValueError: If the secret is not valid JSON
"""
required_fields = ['smtpUser', 'smtpAccessKeyId', 'smtpSecretKey']
# Only do VersionId validation against the stage if a token is passed in
if token:
secret = service_client.get_secret_value(SecretId=arn, VersionId=token, VersionStage=stage)
else:
secret = service_client.get_secret_value(SecretId=arn, VersionStage=stage)
plaintext = secret['SecretString']
secret_dict = json.loads(plaintext)
# Run validations against the secret
for field in required_fields:
if field not in secret_dict:
raise KeyError("%s key is missing from secret JSON" % field)
# Parse and return the secret JSON string
return secret_dict

upload this to s3 inside your cloudformation bucket since we need to reference it for the next cloudformation script.

key-rotation-lambda.yml will generate the rotation service which Secrets Manager will use.

---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
BuildVersion:
Description: Build number
Type: String
Environment:
Description: Deploy Target
Type: String
Product:
Description: Deploy Target
Type: String
Default: "SMTPUserRotation"
NamePrefix:
Description: Name prefix
Type: String
Default: "KeyRotation"
FileLocation:
Description: Lambda function s3 bucket
Type: String
Default: "cf-templates-${region bucket}"
FileName:
Description: Name of the object in s3
Type: String
Default: "aws-key-rotation-lambda.zip"
LambdaHandler:
Description: Lambda handler
Type: String
Default: "aws-key-rotation-lambda.lambda_handler"
LambdaMemoryAllocation:
Description: Memory to be allocated to Lambda function
Type: String
Default: "128"
LambdaRuntime:
Description: Runtime of the Lambda function
Type: String
Default: "python3.6"
LambdaTimeout:
Description: Timeout threshold of the Lambda function
Type: String
Default: "30"
SNSTopicName:
Description: Name of the SNS topic for publishing rotation notifications
Type: String
Default: "key-rotation"
Resources:
IAMRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "${NamePrefix}LambdaRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: !Sub "${NamePrefix}LambdaPolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "iam:ListAccessKeys"
- "iam:DeleteAccessKey"
- "iam:CreateAccessKey"
Resource: !Sub "arn:aws:iam::${AWS::AccountId}:user/smtp*"
- Effect: "Allow"
Action:
- "secretsmanager:GetResourcePolicy"
- "secretsmanager:DescribeSecret"
- "secretsmanager:ListSecretVersionIds"
- "secretsmanager:GetSecretValue"
- "secretsmanager:PutSecretValue"
- "secretsmanager:UpdateSecret"
- "secretsmanager:UpdateSecretVersionStage"
Resource: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:smtp-*"
- Effect: "Allow"
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "*"
- Effect: "Allow"
Action: "secretsmanager:GetRandomPassword"
Resource: "*"
- Effect: "Allow"
Action:
- "sns:Publish"
Resource:
- !Sub "arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${SNSTopicName}"
Lambda:
Type: AWS::Lambda::Function
DependsOn: IAMRole
Properties:
Description:
Fn::Join:
- ""
- - "Build Version "
- !Ref BuildVersion
FunctionName: !Sub "${NamePrefix}LambdaFunction"
Code:
S3Bucket: !Ref FileLocation
S3Key: !Ref FileName
Handler: !Ref LambdaHandler
MemorySize: !Ref LambdaMemoryAllocation
Role:
Fn::GetAtt:
- IAMRole
- 'Arn'
Runtime: !Ref LambdaRuntime
Timeout: !Ref LambdaTimeout
Environment:
Variables:
SECRETS_MANAGER_ENDPOINT: !Sub "https://secretsmanager.${AWS::Region}.amazonaws.com/"
LambdaSecretsManagerPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt 'Lambda.Arn'
Action: "lambda:InvokeFunction"
Principal: "secretsmanager.amazonaws.com"
SNSTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Ref SNSTopicName
mysnspolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: MyTopicPolicy
Version: '2012-10-17'
Statement:
- Sid: ListAllDefaultOwnAccount
Effect: Allow
Principal:
AWS: "*"
Action:
- SNS:Publish
- SNS:RemovePermission
- SNS:SetTopicAttributes
- SNS:DeleteTopic
- SNS:ListSubscriptionsByTopic
- SNS:GetTopicAttributes
- SNS:Receive
- SNS:AddPermission
- SNS:Subscribe
Resource:
- !Ref SNSTopic
Condition:
StringEquals:
AWS:SourceOwner: !Sub '${AWS::AccountId}'
- Sid: SubscribeReceiveOtherAccounts
Effect: Allow
Principal:
AWS: "*"
Action:
- SNS:Subscribe
- SNS:Receive
Resource:
- !Ref SNSTopic
Condition:
StringEquals:
aws:PrincipalOrgID: '${YOUR-ORG-ID from organisations}'
Topics:
- !Ref SNSTopic
Outputs:
LambdaKeyRotationIPARN:
Description: ARN for lambda key rotation
Value: !GetAtt Lambda.Arn
Export:
Name:
Fn::Sub: "${Product}${Environment}LambdaKeyRotationIPARN"
SNSTopc:
Description: ARN for sns topic for rotation notifications
Value: !Ref SNSTopic
Export:
Name:
Fn::Sub: "${Product}${Environment}SNSTopic"


Now that all the building blocks are in place your ready to have a user and new secret created.

Use this next script to generate the iam user and Secrets Manager. Once initialised, it will rotate the secret straight away.

---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
BuildVersion:
Description: Build number
Type: String
Environment:
Description: Environment
Type: String
Service:
Description: Service Name
Type: String
SecretName:
Description: Secret Name
Type: String
SecretDescription:
Description: Secret Description
Type: String
KmsKeyId:
Description: KMS key to use for encryption
Type: String
Default: "" #this is key which allows Secrets Manager and cross account access
IamUserName:
Description: must start with smtp-
Type: String
AllowedPattern: "^smtp-[a-zA-Z0-9-]*$"
CrossAccountList:
Description: comma delimited list of arn's to allow access to the secret
Type: String
AllowedPattern: "^[a-zA-Z0-9-:,]*$"
Default: "arn:aws:iam::111222333444:root,arn:aws:iam::555666777888:root"
LambdaKeyRotationExportName:
Description: export name for the lambda function to rotate the secret
Type: String
Default: "${Product}${Environment}LambdaKeyRotationIPARN"
SnsTopicExportName:
Description: export name of the sns that will notify
Type: String
Default: "${Product}${Environment}SNSTopic"
Resources:
myuser:
Type: AWS::IAM::User
Properties:
Path: "/smtp/"
UserName: !Ref IamUserName
SmtpUserSecret:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Ref SecretName
Description: !Ref SecretDescription
KmsKeyId: !Ref KmsKeyId
SecretString: !Sub
- |
{"smtpUser": "${IamUserName}",
"smtpAccessKeyId": "willbereplaced",
"smtpSecretKey": "willbereplaced",
"smtpUsername": "willbereplaced",
"smtpPassword": "willbereplaced",
"snsTopic": "${SnsTopicExportNameImport}"}
- SnsTopicExportNameImport:
Fn::ImportValue:
!Sub "${SnsTopicExportName}"
Tags:
- Key: "Service"
Value: !Ref Service
# This is a ResourcePolicy resource which attaches a resource policy to the referenced secret.
# The resource policy denies the DeleteSecret action to all principals in the current account.
# It allow attaches any arn's which should have access to the secret.
# NOTE: it should really match the kms key else errors will occur
SmtpUserSecretResourcePolicy:
Type: AWS::SecretsManager::ResourcePolicy
Properties:
SecretId: !Ref SmtpUserSecret
ResourcePolicy:
Version: "2012-10-17"
Statement:
- Effect: "Deny"
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
Action: "secretsmanager:DeleteSecret"
Resource: "*"
- Effect: "Allow"
Principal:
AWS: !Split [ "," , !Ref CrossAccountList ]
Action: "secretsmanager:GetSecretValue"
Resource: "*"
Condition:
ForAnyValue:StringEquals:
secretsmanager:VersionStage: AWSCURRENT
SmtpUserSecretRotationSchedule:
Type: AWS::SecretsManager::RotationSchedule
Properties:
SecretId: !Ref SmtpUserSecret
RotationLambdaARN:
Fn::ImportValue: !Sub "${LambdaKeyRotationExportName}"
RotationRules:
AutomaticallyAfterDays: 30
Outputs:
SmtpUserSecret:
Description: Secrets Manager Secret which was created
Value: !Ref SmtpUserSecret


All it needs now is to have the correct groups/policies added for SES raw email sending.

To this test, get on the cli and change into another account which is not what this is installed into and user the ARN of the secret to call it

aws secretsmanager get-secret-value --secret-id arn:aws:secretsmanager:ap-southeast-2:111222333444:secret:smtp-trial-X4Q0Sp --version-stage AWSCURRENT

With the SNS queue, you can now hook up a lambda function to trigger in the other accounts to go update where ever the stmp credentials are set.

This script will give 60 days per access key and generate a new key every 30 days. So if the system fails to update the first time there is a grace period to change over.