Serverless Webhooks using AWS Lambda - Part 4

This is the fourth and last part of my Serverless Webhooks post. You can find Part 3 here where we built the processor application, Part 2 here where we integrated SQS and Part 1 here where we built the Lambda function handler.

Let’s review again the architecture.

Architecture

In the previous post, we already built the poc-data-processor application and tested locally. It can read the SQS poc-data-feed-queue and update the DynamoDB table poc-data-feed. Our final step is to run this application on AWS. That is we will build the Docker image, push it to ECR, deploy to an ECS container and test in on AWS.

Build the Docker image and push to ECR

Since this is a Spring Boot application, in the Dockerfile we will use the openjdk:8-jdk-alpine image.

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar"]

Build the application

mvn clean package

Build the Docker image

mvn dockerfile:build

Tag the container to your AWS account’s ECR

docker tag adr1/poc-data-processor:latest myaccount.dkr.ecr.myregion.amazonaws.com/poc-data-processor:latest

Your image name may vary. Check your pom.xml if you have a different image prefix. Here I’m using adr1.

    <properties>
        <docker.image.prefix>adr1</docker.image.prefix>
    </properties>
    <plugin>
        <groupId>com.spotify</groupId>
        <artifactId>dockerfile-maven-plugin</artifactId>
        <version>1.3.6</version>
        <configuration>
            <repository>${docker.image.prefix}/${project.artifactId}</repository>
            <buildArgs>
                <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
            </buildArgs>
        </configuration>
    </plugin>

Login to your AWS account

$(aws ecr get-login --no-include-email --region ap-southeast-1)

Push the image to your AWS account’s ECR

docker push myaccount.dkr.ecr.myregion.amazonaws.com/poc-data-processor:latest

Login to the AWS console and verify the image in the ECR.

The Docker image is now in the registry. Next is to configure ECS to run this image.

Configure ECS

In configuring the ECS container, we will create three things - a cluster, a task definition and a service.

The task definition poc-data-processor-task defines how to launch the Docker image poc-data-processor that we just pushed into the ECR. The service poc-data-processor-service manages the workload of running this task. Then the cluster poc-data-processor-cluster which we will define to use the Fargate launch type will take care of the EC2 instances to run these tasks internally.

Read more on the basics of ECS here.

Create the cluster

In ECS, go to Clusters and Create Cluster. We will use Fargate to simplify our setup without worrying much on managing the infrastructure and server details.

Name the cluster as poc-data-processor-cluster and create a new VPC for it. You can use an existing VPC but it is better to separate this PoC setup to isolate it. Later, it would be easier to clean-up too when we tear down this cluster.

Click Create and view the cluster details after creation. Here you will see the networking details that was set-up during the cluster creation like vpc, subnet, internet gateway, etc. You need not worry about these things as Fargate is supposed to take care of these things for you.

Create the task definition

Go to Task Definitions and click Create new Task Definition and choose Fargate.

Take note of the Task Role ecsTaskExecutionRole. We will modify this later to give access to DynamoDB.

For task size, 2GB memory and 1vCPU should be sufficient for our Spring Boot application.

In Container Definitions, click Add container. This is where we define our container and where to get the image. Enter here the image location in the ECR. For the memory, set at least 300MiB that is required by our application.

Leave the rest of the configuration to default. Make sure the Log configuration is ticked so we can monitor in CloudWatch our application.

In Fargate, we wont have access to login to the EC2 server to troubleshoot. So having our application log sent to CloudWatch is important.

After creating the Task Definition, go to IAM and modify the task definition’s role ecsTaskExecutionRole. Add an inline policy to the task process to be able to access DynamoDB. Remember our application needs to read and update the table poc-data-feed.

The inline policy to add is similar to the policy we gave access to our Lambda handler poc-data-feed-handler. You can refer to the first post here and copy the policy from AWSLambdaBasicExecutionRole.

Create the service

Finally, we will create the service poc-data-processor-service to manage running the task definition.

Go back the the cluster poc-data-processor-cluster. In the Services tab, click Create. Here we will specify the task definition and cluster we just created.

The Number of tasks tells Fargate how many tasks instances it will run for this service. For this PoC, we will just specify one. Later, to stop the application, we can set update the service and set this to zero.

Choose the VPC previously created for this and leave the rest to default.

Disable the features we do not need. Set the Load balancer type to None, Service Discovery to disabled, and Auto-scaling to off.

Review and create the service.

It will provision the service and launch the task. Wait for a while until its status is RUNNING.

Full test end-to-end

Similar to the previous posts, we will test this via Postman and send a sample payload.

Go to CloudWatch and Log groups and open /ecs/poc-data-processor-task.

Here you will see similar application logs we did when we tested locally in the previous post. The message is received from SQS, data payload is retrieved from DynamoDB with PENDING status, the data is processed by our task which is running the Docker image data-processor-application and updates the item status to COMPLETED.

You can also verify in DynamoDB the actual item is updated.

That’s it for our ECS setup. Our Docker data processor application is now running on the cloud.

Clean-up

Before we conclude, make sure to tear down the ECS set-up to not incur further costs.

Update the service poc-data-processor-service and change the Number of tasks to 0 (zero). This will stop the running task. After that, delete the cluster.

Summary

This completes our entire Serverless Webhook architecture in AWS. It has been a long series of posts. Hopefully we got a basic understanding of how to set-up a serverless webhook using API Gateway, Lambda, SQS, ECS, and DynamoDB. With this kind of set-up, we can have a cheap data receiver that only runs when data is available. It is easily expandable too by adding more Lambda functions to handle different data sets. If the data is large and requires longer processing time, we have a container task instance to do the heavy workload processing.