A few days ago, I was presenting my topic regarding synapse in ‘Global Azure Bootcamp – 2022’ when one of the participants questioned about how to get notified regarding the status of pipeline activities in a synapse pipeline. After the event, I explored some of the present and effective options that are available and decided to document them with a demo for each method.
Introduction
In Azure Synapse whenever dealing with data ETL/ELT activities, at any point of time the engineer might need to know the state and health of repositories by going through the metrics. The engineer or administrator can accomplish this by setting up notifications within the synapse workgroup for critical metrics which when crossing those thresholds, notification will be sent to the response group for corrective actions.
Create alerts
Once you click on ‘Create new alert’, you will see tabs with multiple options. The ‘Scope’ tab is to select the Synapse workspace if in case you have more than one, here we have only one as listed below.
In the condition tab click on ‘Add condition’ to create new criteria for the alert. Once you click you will see a pop-up with list of metrics and log values which you can select one or more based on your requirement. These are called as ‘Signals’ in azure diagnostics terminology.
Please note that the alert rule in azure is not a free service. $0.10 will be charged for each rule created
Let’s select the signal ‘Pipeline runs ended’ from the list. This will trigger the alert based on pipeline activity or its output, whatever the condition we specify in the coming steps. I will leave that with default hours in the chart period where it will check for the pipeline status in the last 6 hours.
Scroll down to use the next option Split by Dimensions which will let you monitor multiple metric time-series with a single alert rule. A metric time-series is a series of measurements (or "metric values") captured over a period of time. While I am not going to use the time-series monitoring now, just to show you I have selected the ‘Failure Type’ dimension and custom dimension values named Data error & Data Type Mismatch.
Further scrolling down will lead you to ‘Alert Logic’ where you can set the threshold to Static & Dynamic. Since for this demo I have selected pipeline run, I am selecting the Static threshold to alert if the pipeline run ends more than once.
Join the conversation! Your thoughts help the community grow.