Introduction
In this article, we will check how we can copy new and changed files based on the last modification date. The steps have been given below with explanations and screenshots. As of this writing, Azure Data Factory supports only the following file formats, but we can be sure that more formats will be added in the future.
- Avro format
- Binary format
- Delimited text format
- Excel format
- JSON format
- ORC format
- Parquet format
- XML format
Incremental file copy
In data integration, incremental copy of data or files is a common scenario. An incremental copy can be done from the database or files. For copying from the database, we can use watermark or by using CDC (Change data capture) technology. But here we are going to see how to do incremental copy on file level using last modified time from azure blob storage.
Create Storage Account
I have created a simple blob storage account for holding all my files. You can create a new one if you want(refer to my previous blogs on how to create one) or use the one you have currently.

Once inside the blob I have created a container called ‘inputfiles’ and uploaded some files into it.

Once you have all the files ready inside the blob create a new azure data factory instance or use an existing one. I am creating a new one since I have deleted all the resources already. As usual, once created go to the azure data factory studio and go into author mode; the little pencil icon on the left-hand side. If you have any doubts or want to see how to create one, refer to my previous blogs where step-by-step instructions have been given with screenshots.
The next step is to map the blob we created earlier to create a new linked service that can be used to call it inside the pipeline. I have covered it in my previous article hence I will skip it.

Now head back to the author tab to create a new pipeline. Type ‘Copy’ in the search tab and drag it to the canvas; It's with this we are going to perform incremental file copy.

The two important steps are to configure the ‘Source’ and ‘Sink’ (Source and Destination) so that you can copy the files.



















Join the conversation! Your thoughts help the community grow.