AWS RDS Cross Region and Account Backup

Introduction #

Although we already have RDS Automatic backups in place it is prudent to have further backups for Disaster Recovery. The objective here is to copy an automatically generated System Snapshot to another Region and then to another Account. The snapshot is copied as follows…

Account-A, Region-A -> Account-A, Region-B -> Account-B, Region-B

The general approach is to use EventBridge events and rules and some simple Lambda functions.

Alternative Solutions #

There is an official AWS blog post with a similar solution. The main differences between that solution and the one described here are that that solution uses the AWS Backup service and the Backup account is created in the same Organization. We create a new AWS Account outside of the main Organization which allows the Account to qualify for Free Tier.

Create a new AWS Account (Account-B) #

Create a new AWS Account that will be used for backup purposes.

Remember to setup an IAM User and MFA for both the IAM User and the Root User.

Create a Customer Managed Key (Shared Key) #

From Account-B, Region-B:

Go to KMS
Create a Symmetric key with the defaults, (optionally select Multi-Region key to allow for the flexibility to replicate the key to other Regions if needed later).
Select the recently created IAM User as the Administrator.
Under Other AWS accounts add the ID of Account-A - to allow the key to be used by Account-A.

Manually copy the Snapshot (Optional) #

This section is optional as the objective is the automation of this process, however, it provides a quick run-through the process and uses the shared key.

Allow use of the Customer Managed Key by an IAM User in Account-A #

Go to IAM and create a new policy.

The content of this policy is as given in point 4 in this AWS blog post which is reproduced here:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowUseOfTheKey",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": [
        "<ARN-SHARED-KEY>"
      ]
    },
    {
      "Sid": "AllowAttachmentOfPersistentResources",
      "Effect": "Allow",
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": [
        "<ARN-SHARED-KEY>"
      ],
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": true
        }
      }
    }
  ]
}

Next select the IAM User and attach this policy to the User under Add permissions

Manually copy the Snapshot #

From Account-A, Region-A:

Go to RDS Snapshots select a snapshot and select copy.
Select Region-B.
The shared key does not display in the dropdown so the ARN must be inserted manually.
Copy the snapshot.

From Account A, Region-B:

Go to RDS and ensure that the snapshot has been copied there.
Select the Snapshot and Actions, Share snapshot.
Then add the Account-B Account ID.

From Account-B, Region-B:

Go to RDS and ensure that in the Shared with me tab the snapshot is present.
Select the shared key and copy the snapshot.

Automation of Snapshot copy #

Automation, Steps Outline #

As mentioned the approach uses EventBridge and some Lambda functions. The outline of the solution looks as follows.

Account-A, Region-A:
- Rule which matches the system snapshot creation event
  - propagates the event to Region-B
Account-A, Region-B:
- Rule which matches the previously propagated event
  - calls a Lambda function to carry out a cross region snapshot copy
Account-A, Region-B:
- Rule which matches the completion of the cross region snapshot copy
  - calls a Lambda function that does two things
    - share the snapshot with Account-B
    - emits a custom event (to Account-A, Region-B) to signal that the snapshot has been shared
Account-A, Region-B:
- Rule which matches the previously emitted event
  - propagates the event (cross Account) to Account-B, Region-B
Account-B, Region-B:
- Rule which matches the previously propagated event
  - calls a Lambda function to copy the shared snapshot

Automation, Steps Detail #

Here are these steps again in more detail.

1. Rule to match the system snapshot creation event and propagate the event to Region-B #

Create an EventBridge rule with the following pattern:

{
  "source": ["aws.rds"],
  "detail-type": ["RDS DB Snapshot Event"],
  "detail": {
    "SourceArn": [{
      "wildcard": "arn:aws:rds:<REGION-A>:<ACCOUNT-A>:snapshot:rds:<SNAPSHOT-NAME>*"
    }],
    "EventID": ["RDS-EVENT-0091"]
  }
}

Note:

The * wildcard will match the creation date and time.
The second rds: indicates a system snapshot.
A catalog of the EventIDs can be found here AWS RDS Events.

Next setup the Rule Target to propagate the event to Region-B:

Set the Target Type to EventBridge event bus.
Then Event bus in different account or Region.
Add the ARN for the Event bus in Region-B.
This will automatically create the necessary permission policy.
More details can be found in this AWS blog post.

2. Rule to match the previously propagated event and call a Lambda to copy the snapshot #

The pattern is the same as for the previous Rule. The target is a Lambda which copies the recently created snapshot to Region-B:

import json
import boto3

def lambda_handler(event, context):
    print(event)

    source = event['detail']['SourceArn']
    target = event['detail']['SourceIdentifier']
    target = target.replace("rds:", "")

    rds_client = boto3.client('rds', '<TARGET-REGION>')

    response = rds_client.copy_db_snapshot(
        SourceDBSnapshotIdentifier=source,
        TargetDBSnapshotIdentifier=target,
        KmsKeyId="<ARN-SHARED-KEY>")

    print(response)

Note:

rds: must be removed since it is not valid for snapshot names (except for system snapshots created by AWS).
It may be necessary to increase the Lambda runtime from default 3 seconds to around 10 seconds.

The Lambda must have permissions:

to use the shared key - which is the same policy as for the IAM User setup earlier.
to copy the database.

Database permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds:CopyDBSnapshot",
      "Resource": "arn:aws:rds:<REGION-A>:<ACCOUNT-A>:snapshot*"
    }
  ]
}

When the snapshot is copied to Region-B match the across region snapshot copy event:

{
  "source": ["aws.rds"],
  "detail-type": ["RDS DB Snapshot Event"],
  "detail": {
    "SourceArn": [{
      "wildcard": "arn:aws:rds:<REGION-A>:<ACCOUNT-A>:snapshot:<SNAPSHOT-NAME>*"
    }],
    "EventID": ["RDS-EVENT-0060"]
  }
}

The rule target is Lambda function which does two things:

share the snapshot with Account-B.
emits a custom event (to Account-A, Region-B) to signal that the snapshot has been shared.

The Lambda function:

import json
import boto3

def lambda_handler(event, context):
    print(event)

    source_arn = event['detail']['SourceArn']
    source_identifier = event['detail']['SourceIdentifier']
    rds_client = boto3.client('rds', '<REGION-B>')

    rds_client_response = rds_client.modify_db_snapshot_attribute(
        AttributeName='restore',
        DBSnapshotIdentifier=source_identifier,
        ValuesToAdd=['<ACCOUNT-B>',],
    )

    print(rds_client_response)

    events_client = boto3.client('events', '<REGION-B>')

    detail_json = {'SourceIdentifier': source_identifier, 'SourceArn': source_arn}

    events_client_response = events_client.put_events(
                Entries=[
                    {
                        'Source': 'custom.lambda',
                        'DetailType': 'Custom Lambda Snapshot Share Event',
                        'Detail': json.dumps(detail_json),
                    },
                ]
            )

    print(events_client_response)

The Lambda will need permissions to both share the snapshot and emit the event:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds:ModifyDBSnapshotAttribute"
      ],
      "Resource": [
        "arn:aws:rds:<REGION-B>:<ACCOUNT-A>:snapshot:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "events:PutEvents"
      ],
      "Resource": [
        "arn:aws:events:<REGION-B>:<ACCOUNT-A>:event-bus/default"
      ]
    }
  ]
}

4. Rule to matches the snapshot shared custom event and propagates that event to Account-B #

The rule pattern looks as follows:

{
  "source": ["custom.lambda"],
  "detail-type": ["Custom Lambda Snapshot Share Event"],
  "detail": {
    "SourceArn": [{
      "wildcard": "arn:aws:rds:<REGION-B>:<ACCOUNT-A>:snapshot:<SNAPSHOT-NAME>*"
    }]
  }
}

Next setup the Rule target to propagate the event to Account-B:

Set the Target Type to EventBridge event bus.
Then Event bus in different account or Region.
Add the ARN for the Event bus in Account-B, Region-B.
This will automatically create the necessary permission policy.

More details can be found in this AWS User Guide.

Add a rule with a pattern to match the previously emitted snapshot shared custom event:

{
  "source": ["custom.lambda"],
  "detail-type": ["Custom Lambda Snapshot Share Event"],
  "detail": {
    "SourceArn": [{
      "wildcard": "arn:aws:rds:<REGION-B>:<ACCOUNT-A>:snapshot:<SNAPSHOT-NAME>*"
    }]
  }
}

The target is a Lambda similar to that in step 2:

import json
import boto3

def lambda_handler(event, context):
    print(event)

    source = event['detail']['SourceArn']
    target = event['detail']['SourceIdentifier']

    rds_client = boto3.client('rds', '<REGION-B>')

    response = rds_client.copy_db_snapshot(
        SourceDBSnapshotIdentifier=source,
        TargetDBSnapshotIdentifier=target,
        KmsKeyId="<ARN-SHARED-KEY>")

    print(response)