Techskills Zone

Empowering Your Digital Journey

Unraveling Recursive Loops: AWS Lambda’s New Detection Feature

a black and white photo of a pattern

In event-driven applications, AWS Lambda functions are often used to process events from various sources such as Amazon SQS and Amazon SNS. However, due to resource misconfigurations or code defects, a processed event may be sent back to the same service or resource that invoked the Lambda function. This can lead to an unintended recursive loop, resulting in unexpected usage and costs. To address this, AWS Lambda has introduced a feature to detect and stop recursive loops in Lambda functions.

Understanding Recursive Loop Detection

Recursive loop detection in Lambda works by tracking events. Lambda is an event-driven computing service that runs your function code when certain events occur. For example, when an item is added to an Amazon SQS queue or Amazon Simple Notification Service (Amazon SNS) topic, Lambda passes events to your function as JSON objects, which contain information about the change in the system state. When an event causes your function to run, this is called an invocation.

To detect recursive loops, Lambda uses AWS X-Ray tracing headers. When AWS services that support recursive loop detection send events to Lambda, those events are automatically annotated with metadata. When your Lambda function writes one of these events to another supported AWS service using a supported version of an AWS SDK, it updates this metadata. The updated metadata includes a count of the number of times that the event has invoked the function.

A chain of requests is a sequence of Lambda invocations caused by the same triggering event. For example, imagine that an Amazon SQS queue invokes your Lambda function. Your Lambda function then sends the processed event back to the same Amazon SQS queue, which invokes your function again. In this example, each invocation of your function falls in the same chain of requests.

If your function is invoked more than 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you. If you have an on-failure destination or dead-letter queue configured for your function, then Lambda also sends the event from the stopped invocation to your destination or dead-letter queue. Lambda stops only invocations that are part of the same chain of requests. For example, if your function is configured with multiple triggers, then invocations from other triggers aren’t affected.

# Example of a recursive function that could lead to a loop
def recursive_function(event, context):
    # Process the event
    processed_event = process_event(event)

    # Write the processed event back to the same SQS queue
    sqs = boto3.client('sqs')
    sqs.send_message(QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue', MessageBody=json.dumps(processed_event))

Supported AWS Services and SDKs

Lambda can detect only recursive loops that include certain supported AWS services. For recursive loops to be detected, your function must also use one of the supported AWS SDKs.

Supported AWS services include Amazon SQS, Amazon SNS, and AWS Lambda itself. However, when another AWS service such as Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3) forms part of the loop, Lambda can’t currently detect and stop it.

For Lambda to detect recursive loops, your function must use one of the following SDK versions or higher:

  • Node.js: 2.1147.0 (SDK version 2), 3.105.0 (SDK version 3)
  • Python: 1.24.46 (boto3), 1.27.46 (botocore)
  • Java 8 and Java 11: 1.12.200 (SDK version 1), 2.17.135 (SDK version 2)
  • Java 17: 2.20.81
  • .NET: 3.7.293.0
  • Ruby: 3.134.0

Recursive Loop Notifications

When Lambda stops a recursive loop, you receive notifications through the AWS Health Dashboard and through email. You can also use CloudWatch metrics to monitor the number of recursive invocations that Lambda has stopped.

The CloudWatch metric RecursiveInvocationsDropped records the number of function invocations that Lambda has stopped because your function has been invoked more than 16 times in a single chain of requests. Lambda emits this metric as soon as it stops a recursive invocation.

# Example of setting up a CloudWatch alarm for RecursiveInvocationsDropped
cloudwatch = boto3.client('cloudwatch')

cloudwatch.put_metric_alarm(
    AlarmName='RecursiveInvocationsDropped',
    ComparisonOperator='GreaterThanThreshold',
    EvaluationPeriods=1,
    MetricName='RecursiveInvocationsDropped',
    Namespace='AWS/Lambda',
    Period=60,
    Statistic='SampleCount',
    Threshold=1.0,
    ActionsEnabled=True,
    AlarmActions=[
        'arn:aws:sns:us-east-1:123456789012:MyTopic'
    ],
    AlarmDescription='Alarm when more than 1 recursive invocation is dropped',
    Dimensions=[
        {
          'Name': 'FunctionName',
          'Value': 'MyFunction'
        },
    ],
    Unit='Count'
)

Responding to Recursive Loop Detection Notifications

When your function is invoked more than 16 times by the same triggering event, Lambda stops the next function invocation for that event to break the recursive loop. To prevent a reoccurrence of a recursive loop that Lambda has broken, you should:

  1. Reduce your function’s available concurrency to zero, which throttles all future invocations.
  2. Remove or disable the trigger or event source mapping that’s invoking your function.
  3. Identify and fix code defects that write events back to the AWS resource that’s invoking your function. A common source of defects occurs when you use variables to define a function’s event source and target. Check that you’re not using the same value for both variables.

Additionally, if the event source for your Lambda function is an Amazon SQS queue, then consider configuring a dead-letter queue on the source queue. If the event source is an Amazon SNS topic, then consider adding an on-failure destination for your function.

Conclusion

Recursive loops can lead to unexpected charges and resource usage in AWS Lambda. However, with the new recursive loop detection feature, AWS Lambda can now automatically detect and stop these loops, providing a safety net for your serverless applications. This feature is turned on by default and is available in several AWS regions. If you wish to turn off this feature for your AWS account, you can contact AWS Support.

References URLs

https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html
https://docs.aws.amazon.com/lambda/latest/operatorguide/recursive-runaway.html
https://aws.amazon.com/about-aws/whats-new/2023/07/aws-lambda-detects-recursive-loops-lambda-functions/

Unraveling Recursive Loops: AWS Lambda’s New Detection Feature

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top