Amazon Elasticache for Redis with DynamoDB for IoT

In the previous post I created a small application that tracks Amazon IoT Button clicks using AWS Lambda and DynamoDB.

Tracking Amazon IOT Button clicks using AWS Lambda and DynamoDB

There is a simple reporting dashboard as part of the application that uses API Gateway, Lambda, and DynamoDB to retrieve the running counts of SINGLE, DOUBLE, and LONG clicks. The API Gateway endpoint returns a JSON Array that is used to populate a bar chart.

Tracking Amazon IOT Button Clicks

The Python 3 code in the AWS Lambda Function queries DynamoDB, returning a Python 3 List. The Integration Response and Method Response sections of the API Gateway transform the Python List to a JSON Array and return it to the RESTful client.

import boto3
import logging
import os


registered_click_types = os.getenv('registeredClickTypes').split(',')
table = os.getenv('table')

logger = logging.getLogger()
logger.setLevel(logging.INFO)

db = boto3.resource('dynamodb')
table = db.Table(table)


def lambda_handler(event, context):
    counts = []
    
    # querying, but could scan, too.
    for click_type in registered_click_types:
        count = int(table.get_item(
            Key={
                'type': click_type
            }
        )['Item']['counts'])
    
        counts.append(count)

    return counts

If the dashboard polls the endpoint frequently, this can cause a lot of DynamoDB queries and stress on the NoSQL database. If the data does not change frequently, it might be advantageous to add a caching layer using Amazon Elasticache.

Amazon Elasticache for Redis

I hadn’t used Amazon Elasticache before so using it with AWS Lambda was quite the treat. There are 2 caching options for Elasticache, Redis and Memcached. I decided to use Redis as my cache service and spun up a server in my preferred AWS Region.

My goal for this article is to perform simple get and set operations on key-value pairs in Redis, and then later integrate the functionality in the IoT application.

AWS Lambda with Amazon Elasticache and DynamoDB for IoT

There are a couple of challenges with integrating Redis with AWS Lambda, and they have nothing to do with spinning up the managed cache service.

Python Redis Client Dependency

In order to access Redis from Lambda we need to create a deployment package that includes the Redis client. I hadn’t done this before, and it turns out it’s not too difficult. Amazon has some instructions for creating deployment packages with Node, Java, and Python.

I used Python 3 venv to create a virtual environment, pip installed the redis client, and packaged it up along with my code.

Access Redis from AWS Lambda

I wasn’t able to access Redis from Lambda without adding Lambda to the same VPC. I also added it to the same subnet. And, of course, I made sure my Lambda Function’s IAM Role had privileges to use Redis. Seems like a lot if you’re new to AWS Cloud Services, but once you create the function you can make these VPC changes pretty easily in the AWS Console.

AWS Lambda and Elasticache for Redis

Once I successfully learned how to create a Python 3 deployment package to Lambda and configured the Lambda for the same VPC as the Redis server, the following Lambda code worked like a charm to get and set key-value pairs in Redis.

import logging
import os
import redis


logger = logging.getLogger()
logger.setLevel(logging.INFO)

redis_host = os.getenv('redisHost')
redis_port = os.getenv('redisPort')
r = redis.StrictRedis(host=redis_host, port=redis_port, decode_responses=True)


def handle(event, context):
    logger.info('Begin call.')

    key = event['key']
    value = event['value']

    r.set(key, value)

    logger.info(f'The value of {key} in Redis is {r.get(key)}.')

    logger.info('End call.')

I created 2 Python environment variables, called redisHost and redisPort, to hold information about the Elasticache for Redis Server.

The Lambda handler expects the following input, where the values of key and name can be any string.

{
  "key": "name",
  "value": "john doe"
}

I am comfortable using the AWS CLI these days. You can easily invoke the Lambda Function from your favorite terminal. I am invoking the function RedisPlayground by passing in the values above and saving the output to a text file called out.txt.

aws lambda invoke
    --function-name RedisPlayground
    --payload "{\"key\":\"name\",\"john doe\":\"n\"}"
    out.txt

Contents