Iterate through an API that has limits in ADF

Question

I have a ADF pipeline where I have an API call where it returns data in chunks (by 100 max). This pipeline works if you only need 100 users. I been trying to figure a way to call the API again (and increment the count by 100) until the response is empty.

For instance this is the URL for the API (#3) : https://externalAPI.api?$top=100&$skip=@pipeline().parameters.skipCount (begins at 0)

After the forEach activity, I want to check if response is emty, if not, go back to call API activity, then repeat until response is empty. The issue is that there is no way to add an if condition activity that goes back to the API activity if the response is not empty.

One solution I was trying was to make the first three steps as their own pipelines and then call an execute pipeline activity in a second pipeline. I ran into two issues:

1. I'm not sure how to pass the response of the web activity to the next pipeline
2. a ForEach activity is not allowed in a if condition activity as well

I would of thought that the If condition activity would allow you to call a previous activity, but that doesn't seem to be the case. Any suggestions would be appreciated.

Answer 1

You can use Until activity to loop the API call.

1. Create a variable at the pipeline level (ex: repeat) and pass a default value (ex: false).

2. In until loop, add the expression to check the variable value is true.

3. Inside until loop, call the API and update the URL with increment the value in every iteration. Web activity stops running when no response is returned then change the value of repeat variable to true to stop iterating the until loop.