'Pass multiple inputs into Map State in AWS Step Function

I am trying to use AWS Step Functions to trigger operations on a very large S3 file via Lambda. To do this I am invoking a step function with an input that has the S3 key of the file, and byte ranges for that file (each parallel iteration would operate on a different section of the file). The input looks something like

    {
      "job-spec": {
        "file": "some_s3_key",
        "array": [
          "0-100",
          "101-200",
          "201-300", ...
        ]
      }
    }

My Step function is very simple, takes that input and maps it out, however I can't seem to get both the file and the array as input to my lambda. Here is my step function definition

    {
      "Comment": "An example of the Amazon States Language using a map state to process elements of an array with a max concurrency of 2.",
      "StartAt": "Map",
      "States": {
        "Map": {
          "Type": "Map",
          "ItemsPath": "$.job-spec",
          "ResultPath": "$.array",
          "MaxConcurrency": 2,
          "Next": "Final State",
          "Iterator": {
            "StartAt": "My Stage",
            "States": {
              "My Stage": {
                "Type": "Task",
                "Resource": "arn:aws:states:::lambda:invoke",
                "Parameters": {
                  "FunctionName": "arn:aws:lambda:us-east-1:<>:function:some-lambda:$LATEST",
                  "Payload": {
                    "Input.$": "$.array"
                  }
                },
                "End": true
              }
            }
          }
        },
        "Final State": {
          "Type": "Pass",
          "End": true
        }
      }
    }

As written above it complains that that job-spec is not an array for the ItemsPath. If I change that to $.job-spec.array I get the array I'm looking for in my lambda but the key is missing. I tried joining the two together with a | but I hit a limit for how much data I can pass around in Step Functions

Essentially I want each python lambda to get the file key, and one entry from the array

It looks like the Parameters value can be used for this but I can't quite get the syntax right



Solution 1:[1]

Was able to finally get the syntax right.

"ItemsPath": "$.job-spec.array",
"Parameters": {
  "byte_array.$": "$$.Map.Item.Value",
  "file.$": "$.job-spec.file"
},

It seems that Parameters can be used to create custom inputs for each stage. The $$ is accessing the context of the stage and not the actual input. It appears that ItemsPath takes the array and puts it into a context which can be used later.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1