Fundamentals Of Azure Storage

In this article, I will illustrate how we can deliver high scale and low cost solutions with Azure tiered Cloud storage by covering the points given below.

• Introduction to Azure storage.
• Overview of Azure Object (Blob) Storage / Unstructured storage
• Working with Blob storage
• Key concepts of Blob storage
• Architecture
• Demo with all the steps (screenshots)

Attachments

1. Working VS demo of Blob storage project
2. Storage tools

This article is divided into two parts, which are given below.

1. Fundamental of Azure Tiered storage
2. Advance Azure Object (Blob) storage

What is Azure storage and why do we need Azure storage?

Let’s consider the current storage challenges.

Scalability Limits

• Let’s try to predict what storage your business will need over the next 1-2 years. We may need a very long capacity planning lifecycle like 
• What you have to buy, what you have to deploy etc.
• How long would it take you to provision another PB of storage, how quickly you can acquire it, provision it and make it available?
• Even on the small end, if your capacity is full, how long would it take you to provision even a TB and make it available? It means a purchasing event, rack and stack and deploy some gear.
• What are the costs and ramifications, if you have to decommission storage?
  
  Consider the problems you face in the scaling up and scaling down of the traditional storage capacity planning model so far.

Risk

• The data is one of the most important valuable assets to the business, so you can’t put it at risk.
• Golden Rule: You can’t lose, corrupt it, or mess up the data. Make sure it can never be lost or worse, stolen.
• Anytime, you are looking for the alternative solutions; the trustworthiness of the infrastructure solution, which will allow you to deploy your data and protect it.

Cost

• There is an opportunity cost of constraining your business by preventing the ability to generate revenue because you didn’t provision enough capacity.
• Paying the burden of under utilized capacity
• Constant overhead of maintenance, migration, upkeep that doesn’t really add the value or unlock the potential of the data.
• Big events can happen, expansion into a new region or enter a ‘New Geographical Location.' What are the costs, how can we make it seamless?

Business

• All of the above activities and risk factors tax your business and prevent you from focusing on the innovation or the progress.
• The storage should be an asset, which helps you to utilize your data, not something which prevents your business from being agile or utilizing the data and enable it to be competitive.
  
  
  

The key pillars of Azure are given below.

• Limitless scalability
• Trust
• Cloud Economics

These are given in accordance with the intent of enabling the business agility for you and your customers. 

Foundational

• Storage is a fundamental building block for all of Azure Services. Every Azure Service directly or indirectly stores their data in Azure Storage.
• Microsoft’s consumer and enterprise Services are betting their business on Azure storage including XBOX, Skype, Office 365 and more.

The storage Service has a set of capabilities, which is worth talking.

Hyper scale

• We have about 6 billion net new objects added to Azure and we will do almost 60 billion transactions on that data.
• Add that up over a year and the numbers are truly staggering, 100's of trillions of transactions, almost 22EB of data coming in and out of our system and the growth curve is still accelerating.
• Azure is continuing to invest in the infrastructure, constantly taking on new customers every month, as we can see over 120K new subscriptions/ month.

Durable

• Azure has a durable platform. Azure promises to never loose the client data.
• Azure keeps multiple copies of the client data and responds to all the forms of Hardware (HW) failure.
• Constant scrubbing of the data to protect against the corruption or bit rot by constantly reading the data and doing checksums and correcting any copies of the data.

Secure

• Azure now provides encryption at Rest and also client libraries for an Application encryption.

Highly available

• Azure is highly available and it is able to automatically recover.
• Automatic dynamic load balancing. Thus, if you have a set of BLOBs or Table entities and all of a sudden; you have a million visitors ; try to download the object. If Azure storage will notice that traffic increase, then it dynamically scales out and starts serving that object over 100s or 1000s of Servers and then scales back down as the traffic subsides.

Open

• Azure Storage is focused on being the most open platform for the client data regardless of what devices or architectures.
• REST APIs that client can code against, is an open and fully documented and Azure also has built a comprehensive set of the client libraries.
• Azure storage's goal does not make any difference; where the client data is or what type of the Applications, the client is building, the client can do it on a solid platform offering, which depends on it to manage your data at scale.

Hybrid

• Azure storage is the only Cloud provider, which has a great Hybrid story.
• With the release of Windows Server 2016, the new Azure Stack provides a 100% API consistent implementation for all our Services, which allows you to build the Applications and run them On Premise in a local host Service or in Azure public Cloud with zero code changes.
• In addition, Azure storage also offers Azure Services, which bridges your datacenter to Cloud like StorSimple, Azure Backup, Azure Site Recovery.

In addition, Azure Storage is also used by every Azure Service directly or indirectly for the data persistence and by nearly every MS Services including O365 and Skype for Business on the enterprise side and OneDrive, Xbox and Skype on the consumer front

The storage Services have a number of capabilities, which are common across the board. Let’s go through each of them briefly.

Azure Storage Services

• We can break Azure Storage Service broadly into the 2 deployment models, IaaS, where you’re running your own infrastructure or PaaS where you are using higher level services and applications which are developed against the public cloud infrastructure.
• On the Infrastructure side (IaaS), azure provides DISKS and FILES, Disks are single instance Disks which are attached to VMs. There are two flavors, Standard and Premium. And Files are shared storage over SMB. Both are provided as Services to your VMs.
• For PaaS, we have three core objects which are Objects, Tables and Queues.
• Blob storage offers Exabyte scale and the ability to store trillions of objects across different tiers. Our current tiers are Hot storage for higher performance and frequently accessed data and Cool storage for less active & archived data.
• Tables are our massively scalable NoSQL key-value stores for the structured and semi-structured data.
• Queues Service is for reliable intra and inter-Service communications, which are great for decoupling your work flows.

All these Services are built on the common unified distributed storage system. Thus, all of the capabilities which we talked about previously are available to all of these Services.

Storage Durability

• Data Platform Durability is critical, if you don’t offer it, you don’t have a platform ready for the enterprise.
• Azure starts with the Distributed File System which deals with failures, dealing with checksums and data integrity in the background, but another interesting characteristic for us is strong consistency and that’s read after write consistency.
• When you write the data for Azure platform, write it to three different disks on 3 different racks and Azure. Write back when it has been committed to all the three copies.
• If you have an application that is reading and writing you never have this eventual consistency problem that some cloud platforms have.

Replication

• LRS or Locally Redundant Storage, which has 3 replicas, 3 disks on 3 different racks in a single region, which protects against the failures at that level. the client never has to think about the traditional forms of backup such as RAID. You may still need backup for other things but for HW failures, it is no longer necessary.
• Stepping up from there, if you need your data to be resilient against major regional disasters (earthquake, floods), we offer automatic geo-replication to another region 100s of miles away. The replication is async unlike LRS, which is synchronized and strongly consistent. Thus, within a couple of minutes; the data will be replicated to another datacenter.
• The client can query the replication lag, so you know what your RPO is for an Application and use that secondary copy, as needed.
• If you want to use the data for HA, we provide RA-GRS, which is read access. Thus, you get a second DNS endpoint for that second copy.

By going through the above Storage fundamentals, now you will be able to understand what is Azure storage and why we need Azure. It gives us prerequisite fundamental knowledge to advance further in the next article.

Until next time, happy learning.