Invoke Azure Data Factory Pipeline via MSFT Fabric

Problem Statement

Is it possible to Trigger Azure Data Factory (ADF) pipeline from MSFT Data Factory, as there is no out-of-the-box Activity for Azure Data Factory?

Prerequisites

1. Azure Data Factory
2. MSFT Data Factory
3. Organizational Account /Service Principal
   • How to Create a Service Principal
4. APIs used
   • pipelines/createrun
   • pipelines/getrun

Solution

Note. To trigger an ADF pipeline, the Organizational Account / Service Principal must have the below minimum permission on the ADF.

/ Microsoft.DataFactory/factories/pipelines/createrun/action

1. Create a pipeline in MSFT Fabric Data Factory and add the parameters below to the pipeline.

2. Drag and drop the Web activity with the below configurations.

Where create a new connection in web activity with organizational account authentication and appropriate base uri and token audience, click sign in to proceed.

Where

• Base URI: https://management.azure.com
• Token Audience URI: https://management.azure.com

Relative

• URL

  /subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().parameters.DataFactoryName}/pipelines/@{pipeline().parameters.PipelineName}/createRun?api-version=2018-06-01

• Method: POST
• Body: Since we are doing a POST request, we need to define the body and pass a space as the value.

3. The Trigger ADF API call is asynchronous. Hence, we do not know whether the pipeline has actually been completed. The successful execution of the web activity doesn’t mean that the ADF pipeline was a success. To check the status of the Pipeline, add another web activity with the below config.

Use the same connection as created in #2.

• Relative URL

  /subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().parameters.DataFactoryName}/pipelineruns/@{activity('Trigger ADF Pipeline').output.runId}?api-version=2018-06-01

• Method: Get

4. We have to add a polling pattern to periodically check on the status of the trigger until it is complete. We start with an until activity. In the settings of the until loop, we set the expression so that the loop executes until the output of the above web activity is not equal to "Failed" or "Succeeded".

Expression

@or(
    equals(activity('Get Pipeline Details').output.status, 'Failed'),
    equals(activity('Get Pipeline Details').output.status, 'Succeeded')
)

Within Until Activity

Output