.NET for Apache Spark Preview Released

Microsoft announced the preview releases of the .NET for Apache Spark, which provides high performance APIs, making Apache Spark accessible to .NET programmers

Recently, at Spark + AI summit, Microsoft introduced .NET for Apache Spark which provides high-performance APIs for using Spark from C# and F#.
 
Till now, Spark, which is an open source distributed processing engine for analytics on big data sets(used for processing batches of data, real-time streams, machine learning, and ad-hoc query), was accessible through Scala, Java, Python, and R but not .NET.
 
The company said that .NET for Apache Spark will make Apache Spark accessible to .NET programmers across all Spark APIs. Microsoft plans to develop .NET for Apache Spark in the open, as a .NET Foundation member project, along with the Spark and .NET community.
.NET for Apache 
Source: Microsoft 
 
You can use .NET for Apache Spark on Linux, macOS, and Windows as it is compliant with .NET Standard 2.0. It is available by default in Azure HDInsight, and can be installed in Azure Databricks and more.
 
According to the company, the first preview version of .NET for Apache Spark has performed well on the popular TPC-H benchmark.
 
.NET for Apache 
Source: Microsoft 
 
The picture above is showing the per-query performance of .NET for Apache Spark versus Python and Scala. In scenarios where UDF performance is critical such as query 1 where 3B rows of non-string data are passed between the JVM and the CLR .NET for Apache Spark is found 2 times faster than popular analytics language Python.
 
You can visit the official announcement here.