'ADF Copy Pipeline not reading/writing all rows
I have a simple copy pipeline that reads from a CSV file and writes to a Azure SQL database. The pipeline finishes with no errors. On inspection, however, I can see that only 107,506 of the 129,601 rows are actually being read/written. The giveaway that this was not working is that the last row written to the table was not written completely.
The format of the file looks to be intact and formatted correctly. What is causing this issue (timeout issue?), and what options do I have to detect this in the future?
**** EDIT It appears the issue is further up my pipeline. I am using a Logic App to check for new CSV files on an FTP site and then copy these files to Azure Blob Storage. On two occasions, the write of the logic app has cut off the file at 10 MB. This appears to be new as prior to 5/1 I was often exceeding 10MB. Is this in Azure change?
Solution 1:[1]
Check settings in copy activity . I suspect skip incompatible rows is selected in Fault tolerance option.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | AnnuKumari-MSFT |