Category "azure-data-factory"

Issue reading a variable JSON in Azure Data Factory

My pipeline receives the Path and the name of a JSON file. This is part of the flow that I created: The lookup step is used to read the JSON File and later I n

Synapse/Data Factory and Dataverse

Have had a look around but can't see any concrete information. Essentially, if anyone could help it would be great. We are building reporting in the cloud and l

Keep Sink Columns in Copy Activity when Source Columns less than Sink Columns

I have a copy activity in Datafactory that dynamically maps the columns between files in tables A and B. Both tables, A and B are .parquet. Table A has 8 column

Azure Data Factory - Trigger Pipeline Manually Externally

I currently have a Data Pipeline within ADF that pulls data from multiple REST APIs, transforms the data and stores it in an Azure SQL Database and from there i

How set parameters in SQL Server table from Copy Data Activity - Source: XML / Sink: SQL Server Table / Mapping: XML column

I have a question, hopefully someone in the forum could give some help here. I am able to pull data from Soap API call to SQL Server table (xml data type field

How can I change the `Blob path begins with` in the Azure data factory storage event trigger using azure DevOps?

I have set a storage event trigger for the pipelines in the azure data factory Dev environment. the blob path begins with is different in the dev environment an

loop through next url into api url in Azure data factory

I have a an api that contains some data and another api url named "nextapi" I want to loop through each api under api and store the data of each api page to azu

az cli deployment fails with Python error

I am deploying ADF linked templates from ADLS through GitHub Actions and pipeline fails withy generic Python error: ERROR: 'str' object has no attribute 'get' I

Azure Data Factory - Use parameter for typeProperties in storage event trigger

I'm having issues using parameter within a storage event trigger for the typeProperty "blobPathBeginsWith". Per default when using a storage event trigger, the

Flatten two arrays having corresponding values using mapping data flow in azure data factory

I am new to data flows in adf. I have a set of json files in a folder which i would like to parse and flatten two arrays into a csv. The json structure is as fo

POST json data (stored as json files in ADLS) to REST endpoint in Azure Data factory

I have a requirement to POST json data (stored as json files in ADLS) to REST API endpoint. I have to use Azure Data factory for this. I have gone through Micr

Azure ADF Copy Activity with Trailing Column Delimiter

I have a strange source CSV file where it contains a trailing column delimiter at the end of each record just before the carriage return/new line. When ADF is

What is the best way to import Google Analytics data into Azure Blob/Data Lake?

I am trying to import Google Analytics data into Azure Blob or Data Lake storage for analysis or reporting. But I don't see a Google Analytics connector in Azur

ADF-SQL server untrusted domain

I am trying to do a lookup from a sql server table. I am able to connect to the sql server instance via MSSQL management studio, but in ADF I get the below erro

ADF copy activity - ignore the new columns in source without throwing an error

I have a pipeline that copies data from source (dynamics) to SQL server datawarehouse. There is a ForEach activity which iterates over the list of all the table

How to inspect / visualize the current values of MS Azure Data Factory parameters and variables during the pipeline execution

I have ADF pipeline which execute SQL stored procedure (SP). The SQL SP returns output parameters. I would like to visualize the content of the output parameter

Azure Data Factory - copy task using Rest API is only returning first row upon execution

I have a copy task in ADF that is pulling data from a REST API into an Azure SQL Database. I've created the mappings, and pulled in a collection reference as fo

ADF Pipeline format datetime from XML source

I have an Azure Data Factory pipeline with an xml file as a source. The target sink is an Oracle table containing a datetime column. The mapping for most of the

Unable to Publish ADF Storage Event Trigger

I have created storage event trigger in my Azure Data Factory. StorageV2 (general purpose v2) account has been configured with it, If file is place in input con

What is the behavior of ADF schedule triggers when a schedule starts before the previous one ends?

I am trying to contrast and find the exact behavior of ADF schedule triggers and ADF thumbing window triggers when a trigger starts before the previous pipeline