'ADF copy activity - ignore the new columns in source without throwing an error

I have a pipeline that copies data from source (dynamics) to SQL server datawarehouse. There is a ForEach activity which iterates over the list of all the tables and in ADF copy activity the data is copied. Also, the data copy is incremental and that is achieved by using SQL query to load the data incrementally.

However, sometimes new columns are added to the source system but not yet exsist in the destination table. Right now my pipeline stops working and throws an error. Is there a way to skip the newly added columns of the source system in ADF?

sql-serverazureazure-data-factorymicrosoft-dynamics


Solution 1:[1]

You can use the query option in the source and write the query to get the required columns in the select list from the source table.

enter image description here

Or you can edit the mapping in your copy activity and map only the required columns.

enter image description here

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 NiharikaMoola-MT

Related Questions