Working with Azure Data Explorer

Introduction

• Collect log and telemetry data with the support of multiple data streams created by advances software.
• Handle the large amounts of data that can be used for troubleshooting, diagnostics, reporting capabilities.
• Efficient query execution for analysis.

Features

• Store a huge amount of data.
• Support of query language to perform analysis.
• Support of the various types of data: structured, semi-structured, and unstructured.
• Support of multiple Azure services to provide end-to-end data integration solutions.

Working with Azure Data Explorer

• Create database
• Ingest Data
• Query Stored data

Sample Query to execute in Data Explorer

StormEvents
| where StartTime >= datetime(2021-01-01) and StartTime < datetime(2021-07-01)
| where State == "GUJARAT"

• You can also programmatically ingest data into data explorer using PowerShell and C#.
• We can also use power BI with a Data explorer connector to create visualization reports.

Performance of Azure Data Explorer

• Azure Data Explorer uses Kusto Engine V3 which provides advanced features for storage and build a query engine. It provides huge performance features wrt to storing logs and telemetry data plus advances query execution capabilities.
• Azure Data Explorer support various types of file formats including avro,csv,json,multijson,orc etc.

Visualization with Data Explorer

• Apart from providing storage and query capabilities, Azure data explorer also provides visualization capabilities.
• Azure Data Explorer dashboards is a web-based application that runs the query and builds dashboards.
• In addition to the dashboard, it also has native support of integration with other tools,
  
  • Power Bi
  • Excel
  • Grafana
  • Kibana
  • ODBC Connector
  • Qlik

Conclusion

We can use data explorer to store various types of data and perform analytics and ingest data efficiently.