SQS → Lambda Problem With maximumBatchingWindow

Question

Our intention is to trigger a lambda when messages are received in an SQS queue.

• we only want one invocation of the lambda to run at a time (maximum concurrency of one)
• We would like for the lambda to be triggered every time one of the following is true:
  • There are 10,000 messages in the queue
  • Five minutes has passed since the last invocation of the lambda

Our consumer lambda is dealing with an API with limited API calls and strict concurrency limits. The above solution ensures we never encounter concurrency issues and we can batch our calls together, ensuring we never consume too many API calls.

Here is our serverless.yml configuration

functions:
  sqs-consumer:
    name: sqs-consumer
    handler: handlers.consume_handler
    reservedConcurrency: 1 // maximum concurrency of 1
    events:
      - sqs:
          arn: !GetAtt
            - SqsQueue
            - Arn
          batchSize: 10000
          maximumBatchingWindow: 300
    timeout: 900

resources:
  Resources:
    SqsQueue:
      Type: 'AWS::SQS::Queue'
      Properties:
        QueueName: sqs-queue
        VisibilityTimeout: 5400  # 6x greater than the lambda timeout

The above does not give us the desired behavior. We are seeing our lambda triggered every 1 to 3 minutes (instead of 5). It indeed is using batches because we’ll see multiple messages being processed in a single invocation, but with even just one or two messages in the queue at a time it doesn’t wait 5 minutes to trigger the lambda.

Our messages are extremely small, so it's not possible we're coming anywhere close to the 6mb limit.

We would expect the only time the lambda is triggered to be when either 10,000 messages have accumulated in the queue or five minutes have transpired since the previous invocation. Instead we are seeing the lambda invoked anywhere in between every 1 to 3 minutes with a batch size that never even breaks 100, much less 10,000.

The largest batch size I’ve seen it invoke the lambda with so far has been 28, and sometimes with only one message in the queue it’ll invoke the function when it’s only been one minute since the previous invocation.

We would like to avoid using Kinesis, as the volume we’re dealing with truly doesn’t warrant it.