Category "azure-data-factory"

How to embed SQL script in Azure Data Factory Mapping Data Flows Expression Builder

After executing my mapping dataflow I would like to run a clean-up script using the Post SQL Script option of the Sink activity. However I'm not having much su

Error in dataflow plugins.adfprod.AutoResolveIntegrationRuntime.45

I am getting below error while running my dataflow. This dataflow was running fine till yesterday. From today onwards I am getting error like this Operation on

Escape '\' in dynamic content in Azure Data Factory

I am hitting a query through lookup activity on DB2 Database using ADF. Query is: But when I execute the query the '\' count gets double. The output is provide

Expression to check if a given string exist in a output of a script task in ADF

I get following result from my script task, How should I build expression to check if the result contains 'exceed' { "resultSetCount": 1, "recordsAffected": 0,

Data Factory: How to pass JSON as text to Web Body

I have to pass multiple records from a SQL source to the body of a web call in Data Factory in JSON format. eg. {"EmployeeCode":"1234",""FirstName":"Joe","LastN

Issue reading a variable JSON in Azure Data Factory

My pipeline receives the Path and the name of a JSON file. This is part of the flow that I created: The lookup step is used to read the JSON File and later I n

Synapse/Data Factory and Dataverse

Have had a look around but can't see any concrete information. Essentially, if anyone could help it would be great. We are building reporting in the cloud and l

Keep Sink Columns in Copy Activity when Source Columns less than Sink Columns

I have a copy activity in Datafactory that dynamically maps the columns between files in tables A and B. Both tables, A and B are .parquet. Table A has 8 column

Azure Data Factory - Trigger Pipeline Manually Externally

I currently have a Data Pipeline within ADF that pulls data from multiple REST APIs, transforms the data and stores it in an Azure SQL Database and from there i

How set parameters in SQL Server table from Copy Data Activity - Source: XML / Sink: SQL Server Table / Mapping: XML column

I have a question, hopefully someone in the forum could give some help here. I am able to pull data from Soap API call to SQL Server table (xml data type field

How can I change the `Blob path begins with` in the Azure data factory storage event trigger using azure DevOps?

I have set a storage event trigger for the pipelines in the azure data factory Dev environment. the blob path begins with is different in the dev environment an

loop through next url into api url in Azure data factory

I have a an api that contains some data and another api url named "nextapi" I want to loop through each api under api and store the data of each api page to azu

az cli deployment fails with Python error

I am deploying ADF linked templates from ADLS through GitHub Actions and pipeline fails withy generic Python error: ERROR: 'str' object has no attribute 'get' I

Azure Data Factory - Use parameter for typeProperties in storage event trigger

I'm having issues using parameter within a storage event trigger for the typeProperty "blobPathBeginsWith". Per default when using a storage event trigger, the

Flatten two arrays having corresponding values using mapping data flow in azure data factory

I am new to data flows in adf. I have a set of json files in a folder which i would like to parse and flatten two arrays into a csv. The json structure is as fo

POST json data (stored as json files in ADLS) to REST endpoint in Azure Data factory

I have a requirement to POST json data (stored as json files in ADLS) to REST API endpoint. I have to use Azure Data factory for this. I have gone through Micr

Azure ADF Copy Activity with Trailing Column Delimiter

I have a strange source CSV file where it contains a trailing column delimiter at the end of each record just before the carriage return/new line. When ADF is

What is the best way to import Google Analytics data into Azure Blob/Data Lake?

I am trying to import Google Analytics data into Azure Blob or Data Lake storage for analysis or reporting. But I don't see a Google Analytics connector in Azur

ADF-SQL server untrusted domain

I am trying to do a lookup from a sql server table. I am able to connect to the sql server instance via MSSQL management studio, but in ADF I get the below erro

ADF copy activity - ignore the new columns in source without throwing an error

I have a pipeline that copies data from source (dynamics) to SQL server datawarehouse. There is a ForEach activity which iterates over the list of all the table