We use cookies on this site to enhance your user experience. You accept to our cookies if you continue to use this website.

Implement S3 Bucket Lambda triggers in AWS CloudFormation

Lambda Console with S3 trigger

Implement S3 Bucket Lambda triggers in AWS CloudFormation can be quite tricky because of very often circular dependencies or errors like “Unable to validate the following destination configurations” occur. But if you take notice of the following, working with S3 Lambda triggers in CloudFormation will be easier.

  • First, you have to specify a name for the Bucket in the CloudFormation template, this allows you to create policies and permission without worrying about circular dependencies.
  • Secondly, you have to add a DependsOn statement to the Bucket referencing the Lambda Permission, this helps you to fix “Unable to validate the following destination configurations” errors since the bucket will only get created if the Lambda Function and all necessary policies, roles and permissions are in place.

Below you will find a GitHub Gist with a working example which takes care of all tips mentioned above. In this example, created *.txt files are read from a bucket and then deleted.

Read more: https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/

Use or output Amazon MQ Endpoints in AWS CloudFormation

Amazon MQ with ActiveMQ under the hood offers several different protocols, each with its own endpoint. Unfortunately, you can’t reference or output them directly in AWS CloudFormation. To make this possible I use the following variants in a Single Broker setup, where AMQBroker is a CloudFormation resource of type AWS::AmazonMQ::Broker:

and these in an active/standby setup for high availability:

Output Amazon API Gateway Domain Name URL in AWS CloudFormation

Unfortunately, it is currently not possible to output or use the domain name / URL of an Amazon API gateway via Fn::GetAtt in AWS CloudFormation. Therefore I provide the following CloudFormation snippets that enables you to do exaclty this. The snippets use a resource called RestApi of type AWS::ApiGateway::RestApi and a resource Stage of type AWS::ApiGateway::Stage.

SQS Queue as Lambda Trigger in AWS CloudFormation

Lambda Console with SQS trigger

Recently AWS released that the Amazon Simple Queue Service (SQS) is now available as a supported event source for AWS Lambda Functions. You can read the related blog post here:  https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/

Since then I have seen many instructions explaining how to integrate this trigger via AWS Serverless Application Model (AWS SAM). Using this approach I noticed that when rolling out the SAM template via AWS CloudFormation a resource of type AWS::Lambda::EventSourceMapping is created. Since this resource is supported by AWS CloudFormation it should be possible to create the SQS Lambda trigger without SAM.

So I tried it successfully and got the following CloudFormation example template:

One thing to watch out for is that the lambda function timeout is not greater than the visible timeout on the queue. The solution can be tested by sending a test message via the SQS Queue using the SQS Console.

Sending a test message via the SQS Console Sending a test message via the SQS Console

Check if the Lambda Function has been invoked Check if the Lambda Function has been invoked

AWS CloudFormation conditional arrays

Sometimes in CloudFormation a Parameter requires an array, and often an array of variable size is required, determined for example by input parameters. For instance AWS::AmazonMQ::Broker were you need to define an array of SubnetIds. In which either if SINGLE_INSTANCE is selected or if ACTIVE_STANDBY_MULTI_AZ is selected, several ids must be specified.

Using the notation proposed in the documentation this cannot be achieved:

SubnetIds:
        -  !Ref PrivSubnetA
        -  !If [ DeployAmqMultiAzCondition, !Ref PrivSubnetB, ]

But the problem can be solved as follows:

SubnetIds: !If [ DeployAmqMultiAzCondition, [ !Ref PrivSubnetA, !Ref PrivSubnetB ],  [ !Ref PrivSubnetA ]]

Use GitHub source in AWS CodeBuild Project using AWS CloudFormation

AWS CodeBuild with GitHub in North Virigina

I wanted to create an AWS CodeBuild project using AWS CloudFormation, which checks out its sources from GitHub and is triggered via GitHub Webhooks. From these sources, a Node.js application should be built using a self-created docker image stored in ECR (Elastic Container Registry).

Therefore I defined the following template:

At the first try the stack creation failed with the following error message:

No Access token found, please visit AWS CodeBuild console to connect to GitHub (Service: AWSCodeBuild; Status Code: 400; Error Code: InvalidInputException; Request ID: ab458603-6fd4-11e8-9310-ff116e0423f9)

To get rid of this error message it’s necessary to set up the AWS OAuth application to have access to your repositories.

Therefore you have to navigate to the AWS CodeBuild console, create a project and select GitHub as source provider. The project does not need to be saved, it is only important to connect to GitHub.

AWS CodeBuild GitHub AWS CodeBuild GitHub

The next time I tried to deploy the CloudFormation stack, the error message did not appear and the CodeBuild project was created successfully.

CloudFormation CodeBuild CloudFormation CodeBuild

Generate Passwords in AWS CloudFormation Template

Sometimes its necessary to generate random passwords inside a CloudFormation template for instance to secure internet facing applications running on an EC2 or ECS instance. To achieve this you have the possibility to let the user of your Cloud Formation template insert passwords as parameters during the stack creation.

In the following, I will give an example of how to generate passwords in an AWS CloudFormation Template using a Node.js Lambda Function and Custom Resources.

Example Code

Description

We create a Custom CloudFormation Resource and pass a previously created Lambda function as the ServiceToken property. Now on every CloudFormation event (e.g. Create / Update / Delete) on the SampleString resource, the Lambda function will be called. The call contains a so-called ResponseUrl where the Lambda function shall response to. If you understood this procedure the template is really easy to understand. After the creation of the Custom Resource is complete you can use the data stored inside using Fn:GetAtt.

Using the Length property in the Custom Resource you can adjust the password length.

Note: The current version of the script generates a new random password if you performing a stack update which directly involves the Custom Resource (means if you change any parameter or property attached to the Custom Resource). To avoid this you could do a workaround like storing the password in an environment variable of the lambda function and resend it on an update. But normally updates on a custom resource this simple should not happen.

Usage

Inside the AWS Console go to CloudFormation and deploy the example-template.yml. After the stack creation is complete navigate to the Outputs tab and look for the generated password.

AWS Console CloudFormation AWS Console CloudFormation

Rudimentary based on https://github.com/sophos/cloudformation-random-string example implemented in python.

AWS Global Summits 2018

With 100000+ total attendees 2017 in 28 summits over the year, AWS extended their reach. This year AWS is increasing the total amount of summits to 29.

In AWS words AWS Global Summits are:

... free events designed to bring together the cloud computing community to connect, collaborate, and learn about AWS. Summits are held in major cities around the world and attract technologists from all industries, segments, and learning levels who want to learn how the AWS cloud can help them innovate with speed and deliver services with scale, flexibility, and reliability. Attendees will hear from AWS Leaders and Experts, Partners, and Customers. Learn by attending technical breakout sessions, demonstrations, hands-on workshops and labs, and team challenges. Network with AWS Partners and their peers in The HUB—our Partner and Solutions Expo.

Source: AWS 2018

Update: Meanwhile the dates are officially confirmed. Visit: https://aws.amazon.com/summits/ for more.

Currently, the 2018 AWS Global Summits dates and locations are not present at the official Website (URL: https://aws.amazon.com/summits/). But AWS already released an informative pdf file for current and future sponsors which lists dates and locations.  The 2018 summits will be located in the following cities and the most dates are already determined.

EMEA (Europe, the Middle East and Africa)

  • Tel Aviv - March 14
  • Milan - March 27
  • London - May 9 - 10
  • Stockholm - May 16
  • Amsterdam - May 31
  • Berlin - June 6 - 7
  • Paris - June 19
  • Cape Town - TBD
  • Madrid - TBD
  • Bahrain - TBD

APAC (Asia-Pacific)

  • ASEAN - April 4
  • Sydney - April 10 - 12
  • Seoul - April 18 - 19
  • Mumbai - May 10

Greater China

  • Taipei - June 27 -28
  • Shanghai - June 29
  • Hong Kong - July 26
  • Beijing - August 9
  • Shenzen - September TBD

America

  • San Francisco - April 4
  • Mexico City - May 31
  • Sao Paulo - June 21
  • New York City - July 17
  • Chicago - August 2
  • Los Angeles - August 23
  • Atlanta - September 13
  • Toronto - September 20

Japan

  • Tokyo - May 30 - June 1
  • Osaka - June 20

AWS indicates that date and locations subject may change.

aws global summit 2018 map AWS Global Summit 2018 Map

AWS Global Summit 2018 Locations and Dates AWS Global Summit 2018 Locations and Dates

All information and images are based on the following source: https://s3-us-west-2.amazonaws.com/2018globalsummitsponsorship/2018+Global+AWS+Summit+Overview+-+Final.pdf