Azure Data Factory is a cloud-based data integration service which allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and transformation. Using Azure Data Factory, you can create and schedule data-driven workflows (called pipelines) that can ingest data from disparate data stores. It can process and transform the data using compute services, such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics, and Azure Machine Learning.
As we all know, as time goes by, new features are getting pushed to the Azure Services. A good example of this is late 2018 when Azure Function was added in ADF (Azure Data Factory), assuming a designer has designed a pipeline with a combination of other services, like ALA (Azure Logic Apps). After a few months, ADF got the same feature as what ALA is doing today. The questions are:
- How can we replace the ALA with new activities in ADF?
- How can we change/add new features with a minimal amount of changes?
- How can we reuse/replace a pipeline?
Well, the answers are -
- ADF Framework
- Master Pipeline Framework
- Parent Pipeline Framework
- Child Pipeline Framework.
This video contains an introduction to the following by experts:
What Azure Data Factory is, the design of Pipeline Framework, the design of ADF Framework, and an overview of Master Pipelines.