'Can an ADF pipeline change its Concurrency number based on a calculated condition?
Background
I have an ADF pipeline that takes jobs from a “3rd party” queue (ueing a REST call), completes jobs and marks the queued message as complete. If the queue is empty, the pipeline ends. Otherwise, the pipeline loops until the queue is empty.
I can scale the pipeline by creating multiple “Scheduled” triggers running every one minute. I can control the number of the pipeline instances running by changing its Concurrency numbers.
Question
Is there any way the ADF pipeline changes its Concurrency number based on a calculated condition such as the queue length?
Solution 1:[1]
Currently you can give only numeric hardcoded values in concurrency settings and there is no option to pass the value dynamically.
Reference: Azure Data Factory Pipeline Concurrency
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | MayankAvasthi-MT |