Introducing U-SQL- Make Big Data Processing Easy

Microsoft recently announced a new feature for Azure Services, Azure Data Lake Services for analytics in the Microsoft Azure cloud services. It is having a large scale repository and also a default service YARN that provide a benefit for analysing the large scale data to a developer or DBA and also for the data scientists. YARN is Apache Hadoop NextGen MapReduce known as MapReduce 2.0 (MRv2) or called YARN.

Azure Data Lake have a managed way to manage all Hadoop, Spark and HBase Services. Azure Data Lake use U-SQL. It is a language that explores the benefits of SQL and is scalable and provides a distributed query capabilities for a developer and DBA those are currently working with Big Data. Also, it provides you to efficiently analyse your data in the store and across relational stores such as Microsoft SQL Azure Database.



Benefits of Data Lake

Batch, Real-time, and interactive analytics made it easy to:

• Store and analyze data of any kind and size.
• Develop faster, debug and optimize smarter.
• Interactively explore patterns in your data.
• No learning curves—use U-SQL, Spark, Hive, HBase, and Storm.
• Managed and supported with an enterprise-grade SLA.
• Dynamically scales to match your business priorities.
• Enterprise-grade security with Azure Active Directory.
• Built on YARN, designed for the cloud.

Why use U-SQL?

U-SQL is a large scalable Language and if we see the behaviour of Big Data analytics we can have several requirements. Here are the major requirements as per the MSDN blog.

• Process any type of data From analysing BotNet attack patterns from security logs to extracting features from images and videos for machine learning, the language needs to enable you to work on any data.
  
  
• Use custom code easily to express your complex, often proprietary business algorithms. The example scenarios above may all require custom processing that is often not easily expressed in standard query languages, ranging from user defined functions, to custom input and output formats.
  
  
• Scale efficiently to any size of data without you focusing on scale-out topologies, plumbing code, or limitations of a specific distributed infrastructure.