Working with Azure Data Explorer

Introduction

  • Collect log and telemetry data with the support of multiple data streams created by advances software.
  • Handle the large amounts of data that can be used for troubleshooting, diagnostics, reporting capabilities.
  • Efficient query execution for analysis.

Features

  • Store a huge amount of data.
  • Support of query language to perform analysis.
  • Support of the various types of data: structured, semi-structured, and unstructured.
  • Support of multiple Azure services to provide end-to-end data integration solutions.

Working with Azure Data Explorer

  • Create database
  • Ingest Data
  • Query Stored data

Sample Query to execute in Data Explorer

StormEvents
| where StartTime >= datetime(2021-01-01) and StartTime < datetime(2021-07-01)
| where State == "GUJARAT"

  • You can also programmatically ingest data into data explorer using PowerShell and C#.
  • We can also use power BI with a Data explorer connector to create visualization reports.

Performance of Azure Data Explorer

  • Azure Data Explorer uses Kusto Engine V3 which provides advanced features for storage and build a query engine. It provides huge performance features wrt to storing logs and telemetry data plus advances query execution capabilities.
  • Azure Data Explorer support various types of file formats including avro,csv,json,multijson,orc etc.

Visualization with Data Explorer

  • Apart from providing storage and query capabilities, Azure data explorer also provides visualization capabilities.
  • Azure Data Explorer dashboards is a web-based application that runs the query and builds dashboards.
  • In addition to the dashboard, it also has native support of integration with other tools,
     
    • Power Bi
    • Excel
    • Grafana
    • Kibana
    • ODBC Connector
    • Qlik

Conclusion

We can use data explorer to store various types of data and perform analytics and ingest data efficiently.