Category "azure-data-factory-2"

creating metadata of table in Sink/Target in Data Factory automatically

Is it possible on Azure Data Factory to create the table definition in the target/sink database. SSIS used to create the metadata automatically based on the Sou

I can’t see the trigger I’ve created in arm-template-parameters-definition.json

The problem is, I’ve created in the arm-template-parameters-definition.json a trigger with the name “Test” this due the fact that I want to pa

Azure Data Factory JSON Syntax

In ADF, I have a copy activity. The data source is the response body from REST API POST request. The sink is a SQL table. The problem I'm having is that even th

Azure Storage Account file details in a table in databricks

I am loading data via pipelines in ADLS gen2 container. Now I want to create a table that has details that when the pipeline start running and then completed. l

Flatten two arrays having corresponding values using mapping data flow in azure data factory

I am new to data flows in adf. I have a set of json files in a folder which i would like to parse and flatten two arrays into a csv. The json structure is as fo

Azure Data Factory 2, Data flow with derived column, how to set a timestamp to null

I have a Azure DataFactory V2 pipeline that executes a data flow. The Azure Sql source table have a Date not null column, DateReported. This is imported from an

Linked Service on Azure Data Factory V2 (on active directory 1) to an Azure PostgreSQL server (on active directory 2)

I'm trying to create a linked service on Azure Data Factory V2 (from my personal subscription) to an Azure PostgreSQL database hosted by company A (on a differe

Unable to Publish ADF Storage Event Trigger

I have created storage event trigger in my Azure Data Factory. StorageV2 (general purpose v2) account has been configured with it, If file is place in input con

What is the behavior of ADF schedule triggers when a schedule starts before the previous one ends?

I am trying to contrast and find the exact behavior of ADF schedule triggers and ADF thumbing window triggers when a trigger starts before the previous pipeline

Can an ADF pipeline change its Concurrency number based on a calculated condition?

Background I have an ADF pipeline that takes jobs from a “3rd party” queue (ueing a REST call), completes jobs and marks the queued message as compl

How to use Wildcard Filenames in Azure Data Factory SFTP?

I am using Data Factory V2 and have a dataset created that is located in a third-party SFTP. The SFTP uses a SSH key and password. I was successful with creatin

Azure Data Factory (ADF) - Parse/Flatten XML file - Get content of all elements match wildcard criteria and sitting in different segments in Hierarchy

I have XML file something like below and need to get all complex elements having different names but all ends with "_KEYS" and they are part of different segmen

ADF: manage token expiration in copy data activity

I am trying to copy data from a REST API into an Azure SQL Database table. The copy data activity uses pagination and can run longer than 10 minutes after which

How to prevent data factory from running out Cosmos DB RU/s

Description When I copy data from Storage to CosmosDB by Azure data factory, the Cosmos DB RU/s is full and nobody can use it during this time. I want other ope

Other Categories