'Pass multiple inputs into Map State in AWS Step Function
I am trying to use AWS Step Functions to trigger operations on a very large S3 file via Lambda. To do this I am invoking a step function with an input that has the S3 key of the file, and byte ranges for that file (each parallel iteration would operate on a different section of the file). The input looks something like
{
"job-spec": {
"file": "some_s3_key",
"array": [
"0-100",
"101-200",
"201-300", ...
]
}
}
My Step function is very simple, takes that input and maps it out, however I can't seem to get both the file and the array as input to my lambda. Here is my step function definition
{
"Comment": "An example of the Amazon States Language using a map state to process elements of an array with a max concurrency of 2.",
"StartAt": "Map",
"States": {
"Map": {
"Type": "Map",
"ItemsPath": "$.job-spec",
"ResultPath": "$.array",
"MaxConcurrency": 2,
"Next": "Final State",
"Iterator": {
"StartAt": "My Stage",
"States": {
"My Stage": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:<>:function:some-lambda:$LATEST",
"Payload": {
"Input.$": "$.array"
}
},
"End": true
}
}
}
},
"Final State": {
"Type": "Pass",
"End": true
}
}
}
As written above it complains that that job-spec
is not an array for the ItemsPath
. If I change that to $.job-spec.array
I get the array
I'm looking for in my lambda but the key
is missing. I tried joining the two together with a |
but I hit a limit for how much data I can pass around in Step Functions
Essentially I want each python lambda to get the file key, and one entry from the array
It looks like the Parameters
value can be used for this but I can't quite get the syntax right
Solution 1:[1]
Was able to finally get the syntax right.
"ItemsPath": "$.job-spec.array",
"Parameters": {
"byte_array.$": "$$.Map.Item.Value",
"file.$": "$.job-spec.file"
},
It seems that Parameters
can be used to create custom inputs for each stage. The $$
is accessing the context of the stage and not the actual input. It appears that ItemsPath
takes the array and puts it into a context which can be used later.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 |