Configuring An ADF Pipeline Activity Output To File In ADLS

Introduction

At an enterprise level every project schedule and runs multiple ADF pipelines but tracking their outcomes in ADF studio is a cumbersome process currently. After your pipeline activity has thrown some error, you will have to track them one by one until you find the failed one and click on its output button to see the error. In this article, we will see the practical approach to configure the output of a pipeline activity to a file in Azure blob storage.

Configuring an ADF Pipeline activity Output to File in ADLS

Open azure data factory studio from azure portal and create a demo pipeline. Once created drag an ‘Execute Pipeline' activity onto the canvas inside which we are going to configure the main activity.

I have created a new pipeline with failed activity just for this demo, which would be invoked inside our first pipeline.

After this, I must capture the error message that gets thrown in the ‘fail activity’ into a variable, hence I will pull the ‘set variable’ activity to the canvas. After this connect both the activities but from "On failed" action from the Exec pipeline1.

For the set variable activity add a new variable name and type as string.

The value parameter is where we will capture the output of the error that we are going to receive from the pipeline. Initially, I added the last part of the below dynamic content as output.error.Message which was causing me error. I have provided the exact query which you can use and save time from fixing the error and code correction.

@activity('Execute Pipeline1').error.Message

Now when I run the pipeline it captures the error output message in the variable that we have created.

Now all set, now let’s move the error messages onto the text file in adls as per the intention of this article. For this, I am going to add one more activity, the Copy activity at the end of our pipeline.

This will copy the variable that is held in the set variable activity which will run prior to this into a text file in adls storage. In the source, I have pointed to a dummy CSV file with only column headers so that I can pass the variable along in the CSV file to the text file but filter only the variable column in it. The next steps will explain how this can be achieved.

In the source just add an additional column(which is going to be written in the destination text file) and add the variable name that we set in the previous step.

Sink will be the location where you want the output to be stored.

Now in the mapping tab click on import schemas and import all the columns, you will see the columns from source file including the additional column which you have set in source tab. Now go ahead and delete all the columns from source and keep only the variable column which is going to write the output onto the text file.

Once you have deleted, it will look like below

This pipeline is all set to be run, let's go ahead and trigger it.

We could see the first step shows failed which is expected behaviour as we have set up a failure activity inside it. Let’s check the destination adls storage for the file that has been created and the contents.

The file has successfully been written to a txt file as BusinessData.txt in my destination folder with the right error message.

Summary

Today we learned how the ADF pipeline error message can be written over to a text file in a storage location so we can save time on clicking each file for activity outputs.