In our AWS cloud native application development projects we often use native-cloud technology like AWS Lambda and AWS ECS Fargate to accelerate deployment and minimize costs. The power of both products is off course the (low) pay-per-use model but even better the low total operating expenditures (OPEX) in the lifecycle management of the application.
What we noticed in the initial cloud native applications we maintained from DevOps perspective is that quite often this meant two other components became relative expensive. First monitoring, logging, tracing and alerting in a distributed application (that’s for another blog) and the relational databases.
Why ? Because auto-scaling on the different RDS types is quite often not that advanced, leaving you with a scale-to-max setup on environments for the performance tests and production.
With the announcement of Aurora Serverless we started using this for a new project and later one started migrating existing applications. Why ? Because it has some unique features on the data layer
✔︎ Easy scaling (less OPS burden)
✔︎ Pay-per-use (quite unique as said)
✔︎ Accessible over HTTP (Data API)
✔︎ Authentication & authorization using IAM roles rather than database roles
An infra-as-code example
So below is an example CloudFormation template how to configure the database. Because if you do it, you do it right. We first generate both the master and a system user for Lambda and store it in SecretsManager.
Resources:
# SecretsManager
# Generate the master user and store it
SecretsManagerSecretMaster:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub 't10/${ENV}/rds/t10master'
Description: !Sub 't10/${ENV}/rds/t10master'
GenerateSecretString:
SecretStringTemplate: '{"username": "t10master"}'
GenerateStringKey: 'password'
PasswordLength: 15
ExcludeCharacters: '"@/'
SecretsManagerSecretTargetAttachmentMaster:
Type: AWS::SecretsManager::SecretTargetAttachment
Properties:
SecretId: !Ref SecretsManagerSecretMaster
TargetId: !Ref AuroraCluster
TargetType: AWS::RDS::DBCluster
# LAMBDA user: manual create it within the Database using the master
SecretsManagerSecretLambda:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub 't10/${ENV}/rds/t10lambda'
Description: !Sub 't10/${ENV}/rds/t10lambda'
GenerateSecretString:
SecretStringTemplate: '{"username": "t10lambda"}'
GenerateStringKey: 'password'
PasswordLength: 15
ExcludeCharacters: '"@/<>'
SecretsManagerSecretTargetAttachmentLambda:
Type: AWS::SecretsManager::SecretTargetAttachment
Properties:
SecretId: !Ref SecretsManagerSecretLambda
TargetId: !Ref AuroraCluster
TargetType: AWS::RDS::DBCluster
SecretsManagerRotationScheduleLambda:
Type: AWS::SecretsManager::RotationSchedule
Properties:
SecretId: !Ref SecretsManagerSecretLambda
RotationLambdaARN: 'arn:aws:lambda:eu-west-1:xxxx:function:SecretsManagermysql-rotation' #generic
RotationRules:
AutomaticallyAfterDays: 30
The next segment will then create the security group, parameter group and the Aurora serverless database itself.
Resources:
AuroraSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: !Sub 't10-${ENV}-sg-aurora'
GroupName: !Sub 't10-${ENV}-sg-aurora'
VpcId: vpc-123456
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 3306
ToPort: 3306
CidrIp: 10.0.0.0/8
SecurityGroupEgress:
- IpProtocol: '-1'
CidrIp: 0.0.0.0/0
Tags:
- Key: Name
Value: !Sub 't10-${ENV}-sg-aurora'
AuroraCluster:
Type: AWS::RDS::DBCluster
Properties:
BackupRetentionPeriod: 28
DBClusterParameterGroupName: !Ref DBClusterParameterGroup
Engine: aurora
EngineVersion: 5.6.10a
EngineMode: serverless
ScalingConfiguration:
AutoPause: false
MinCapacity: 1
MaxCapacity: 64
# SecondsUntilAutoPause: 86400
StorageEncrypted: true
MasterUsername: !If
- UseDbSnapshot
- !Ref 'AWS::NoValue'
- t10master
MasterUserPassword: !If
- UseDbSnapshot
- !Ref 'AWS::NoValue'
- !Join ['', ['{{resolve:secretsmanager:', !Ref SecretManagerSecretMaster, ':SecretString:password}}' ]]
DatabaseName: !If
- UseDbSnapshot
- !Ref 'AWS::NoValue'
- !Sub 't10${ENV}'
DBClusterIdentifier: !Sub 't10-${ENV}-dbcluster'
Port: 3306
DBSubnetGroupName: !Sub 't10-${ENV}-dbsubnetgroup'
SnapshotIdentifier: !If
- UseDbSnapshot
- !Ref DBSnapshotIdentifier
- !Ref 'AWS::NoValue'
DeletionProtection: true
VpcSecurityGroupIds:
- !Ref auroraSecurityGroup
DBClusterParameterGroup:
Type: AWS::RDS::DBClusterParameterGroup
Properties:
Description: !Sub 't10-${ENV}-parametergroup'
Family: aurora5.6
Parameters:
event_scheduler: 'ON'
Tags:
- Key: Name
Value: !Sub 't10-${ENV}-parametergroup'
Going into details
We use a common AWS CloudFormation mechanism here to determine if we need to restore from an earlier snapshot by checking if the parameter DBSnapshotIdentifier is passed as input.
Conditions:
UseDbSnapshot: !Not
- !Equals
- !Ref DBSnapshotIdentifier
- ''
The power of Aurora Serverless is in this segment. By setting the ScalingConfiguration we let our database auto-scale between a minimum and maximum capacity. So no complex auto-scaling configuration with events, no provisioning of RDS instance types and clusters. This is it. In the below example the AutoPause feature is disabled which means the database is always there in the least minimum capacity (1 in the example). If you enable it the RDS will stop completely after 0 connection during the SecondsUntilAutoPause (set to the 24 hours maximum here). This is nice for PoC and personal accounts (to save money) but bare in mind that starting the database can take up to 60 seconds (personal experience).
ScalingConfiguration:
AutoPause: false
MinCapacity: 1
MaxCapacity: 64
# SecondsUntilAutoPause: 86400
The scaling configuration result can then be something like the image below. Since it’s pay-per-use operational cost can decrease drastically as we have seen for our cloud native applications.
Hope it helps!
References:
- Post image leant from AWS Geek Jerry Hargrove (awesome website)
- Aurora Design Considerations @ AWS
