Be Sure With Azure (.NET): Blob Storage

I should point out that you can either step through the superficial structure one hierarchy at a time or combine them as in “family/kids”. Using the container ListBlobs method on say a CloudBlobDirectory for just “family” would show us both the “kids” and “relatives”.

This would be a good time to point out that utilizing a method such as ListBlobs on a CloudBlobDirectory will be showing an IEnumerable<IListBlobItem>, but in this case it is not providing a list of individual metadata about actual BLOBs, but about the folder structure.

Tip: It is important to note that Azure BLOB Storage was not created with a hierarchical structure in mind. That being said, we generally will find ourselves in trouble when we attempt to make something do something that was not its original purpose. I would advise making use of containers as that hard boundary for grouping BLOBs by their relativity. Because at the end of the day, the superficial hierarchical folder structure is simply in the naming of the BLOB.

BLOB Snapshots (Backups)

Azure BLOB Storage provides a mechanism for creating backups of BLOBs as a snapshot. Snapshots are read-only copies of a BLOB at a specific time.
It's a good time to point out that in many of these Storage Client SDK methods that we have looked at in this article, allow for providing optional parameters. I have opted to keep this simple and look at the default use cases for most of them. Take for example, CreateSnapshot that also takes a collection object of string key/value pairs, such as a dictionary for specifying metadata for a BLOB snapshot at the time of creation.

As pointed out before, creating snapshots creates a small overhead to the act of deleting a BLOB if snapshots exists. Attempting to delete the preceding BLOB as in the following:
Would throw an HTTP 409 Conflict status code. Therefore, to delete the snapshot we would need to use the DeleteSnapshotsOption enumeration, IncludeSnapshots or None in order to delete the BLOB (None will throw an error if a snapshot does exists):
We can delete only the snapshots by using the DeleteSnapshots options. The CreateSnapshot returns a BLOB reference to the snapshot. This snapshot has a property Time that will return when the snapshot was taken. Using this Time as well as a reference to the snapshot, you can replace the actual BLOB.

Where Am I?

Working with the Azure Storage Client SDK we have been working with containers and BLOBs by acquiring references and interfacing with container and BLOB objects. But you need to remember that the provided Azure API is a REST API and under the covers, the Storage Client SDK is handling making those HTTP/HTTPS requests to Azure. With that being said, the one aspect of all storage resources that we have been shield from by the Storage Client SDK is it's resource location.

One of the most important aspects of the container and BLOB is its associated storage URI. Containers and BLOBs have both a primary and secondary storage URI that defines where the BLOB or container resource can be found and retrieved. In addition, both containers and BLOBs have a URI property that only returns the primary location URI.

At the beginning of this article we talked about the concept of namespaces and primarily focused on the primary namespace involving the storage account. The container and BLOB primary and secondary URI is made up of the storage account namespace and its own relative address.

While we have mainly been setting up containers, uploading, downloading and manipulating BLOBs, the use of their URI really comes into play when a consumer, consumes the storage resource outside the scope of the Storage Client SDK such as displaying an image on a webpage or an interoperation by a third-party application just to name a couple of the infinite scenarios.

Asynchronous Operations

When looking through the CloudBlobContainer and CloudBlockBlob available methods, you will be inundated with a slew of options for the same operation. However, a vast majority are to support the older Asynchronous Programming Model (APM) with their Begin… and End…operation for every operation that supports asynchronous execution.

In regards to utilizing the asynchronous operations that the Storage Client SDK provides, the newer Task-based Asynchronous Pattern (TAP) based methods will be the asynchronous methods you will want to leverage. Following the TAP convention, these will be methods ending in Async such as UploadFromFileAsync. The swath of asynchronous methods that you see that support other asynchronous patterns are to support backward compatibility for the most part.

Since this is not an article on asynchronous programming, I will leave you with a couple resources for working with Tasks such as the MSDN page on TAP and working with aysnc and await keywords.

Concurrency

When we are dealing with multi-tenant applications that paint a high probability scenario where the same data could be updated simultaneously by multiple users, we need a way to manage those concurrent updates. Azure BLOB Storage provides the following two concurrency control mechanisms for doing just that:

• Optimistic Concurrency (via the ETag property)
• Pessimistic Concurrency (via leases)

Optimistic Concurrency

In an optimistic concurrency model, resources are not locked and inaccessible to the clients. The system will assume that though conflicts are possible, they will be rare. Instead of locking a resource ever time it is in use, the system will look for indicators that multiple tenants actually did attempt to update the same resource at the same time. Azure BLOB Storage provides a optimistic concurrency control through the ETag property. This is available on both the BLOB as well as the container.

Every time an update is made to a BLOB or container you can see this value increase. Essentially, if an update is attempted against a container or BLOB that is out of order it will be rejected and an exception will be thrown.

We can utilize this concurrency control by providing an AccessCondition object on our update attempt. You will find a vast majority of update operations in the Storage Client SDK has an optional AccessCondition parameter.

We are setting the condition in which a successful update can occur. There are other AccessCondition properties that allow you to further control a success update as well. If a failed attempt has occurred, a 314 Not Modified HTTP Status code will be returned.

Pessimistic Concurrency

Pessimistic Concurrency is called pessimistic because it assumes the worst case. It assumes that two tenants will attempt to update the same resource at the same time. Therefore it will allow a lock to be placed on the resource by the first tenant, barring all further updates by other tenants.

Azure Storage's form of Pessimistic Concurrency is through the form of Leases. Both containers and BLOBs offer lease options. There is a difference between both leases where container leases are restricted to delete operations while BLOB leases are for both write and delete operations.

Container Leases

These container leases are strictly delete leases to ensure that the container is not deleted until either the specified length of time designated when the lease was obtained has expired or the lease has been released (not just broken). The time span that can be set for a lease is either infinitely or 15-60 seconds. The only caveat is if the lease ID is provided with the delete request.
But you can either pass in an AccessCondition object with the specified Lease ID or release the lease before calling delete as in the following:
or:
We have seen how to Acquire and Release leases, but you can also Change, Renew and Break container leases. As a quick explanation:

• ChangeLease: will allow you to change the ID of the lease in question. This is useful when you want to transfer a lease to another party but maintain control over the lease
• RenewLease: does exactly as you would expect and allows you to renew a current lease. In previous versions, this was able to be conducted on expired leases.
• ReleaseLease: This allows the containers current lease to be released and the container to be free for another client to acquire an exclusive lease. It is important to understand the difference between ReleaseLease and BreakLease. Where BreakLease ends the lease, but ensures that another lease cannot be acquired by another client on the container until the current lease time period has expired.

BLOB Leases

BLOB leases establish a lock on BLOBs for both write and delete operations. The time span that can be set for a lease is either infinitely or 15-60 seconds just as with Container leases. This is the primary difference between container and BLOB leases where container leases are for locks strictly on delete operations. Because of the additional write lock that BLOB leases manage, an overhead is put on a number of write related operations requiring the Lease ID. These operations include:

• BLOB upload operations
• Setting BLOB Metadata
• Setting BLOB properties
• deleting BLOBs
• copying BLOBs

Therefore, acquiring a lease on an existing BLOB and attempting to update will result in a returned 412 HTTP status code:

You will need to provide an AccessCondition object specifying the current lease ID:
This same procedure would apply to all the preceding listed write operations when a lease exists on a BLOB. As far as delete locks by the BLOB lease, you can reference the "Container Leases” section under Concurrency for deleting BLOBs that have an active lease as the same mechanisms apply, but only within the context of a BLOB.

Tip: A gotcha that you need to be aware of regarding leases for both BLOBs and containers, they have a limited range of time that you can set for the lease as specified earlier and if an attempt to set a timespan that is outside of the 15 – 60 seconds (other than infinite), the service will return a 400 Bad Request HTTP status code. Although, the status code makes sense it still can be a bit vague as to “what” part of the request was bad (incorrect).

Security

Azure Storage security comes in a few different forms. But at the end of the day most of it would boil down to access control. The first form of security that is the least complex is the simplicity of setting the BLOB Container public access permissions.

Container Permissions

You will have noticed that in the CreateIfNotExists operation for containers, we are passing a BlobContainerPublicAccessType enumeration. This is the public access setting for the container itself.
The various types breaks down as follows:

• Off: No public access and only the owner (someone with access keys) can read resources from the container
• BLOB: Public access to a specific BLOB if a anonymous client knows the direct BLOB URI
• Container: Specifies anonymous clients can retrieve information on the resources of the container, utilizing container methods such as ListBlobs to retrieve the containers resources.

That's all there is to it. The only other piece of information would be to use the Principle of Least Privilege and provide the minimal access to your container as possible for consumers to be able to operate.

Shared Access Signatures

We have already discussed that in order to access BLOB storage resources we need to have access to the Storage Access Keys. So if we were to provide additional clients direct access to BLOB storage resources, they would also need access to our Storage Access Keys. Yes, the same Storage Access Keys we defined earlier as “Keys to the Kingdom”. So why on earth would we want to do that? We don't give the keys to our house when someone asks for our home Address do we? This is where Shared Access Signatures can make it possible to provide clients access to BLOB storage resources without giving out our Storage Access Keys.

A Shared Access Signature (SAS) provide a granular level of access control to specific resources. Shared Access Signatures allow you to specify what access rights a consumer has for a specific resource for a given amount time. They can be created for containers, BLOBs, tables and queues. But, in this article we will strictly look at SASs for BLOB Storage. In the next up coming “Be Sure with Azure” on Table Storage, we'll look at how to utilize Shared Access Signatures for tables.

So what is a Shared Access Signature? Simply put, a SAS is a query string with a set of parameters that specifies the access rights and length of access. This query string accompanies a BLOB storage resource's URI. As we have already discussed, all BLOB storage resources (containers and BLOBs) have an associated primary and secondary URI for directly accessing that resources. Therefore, when the generated SAS is appended to the URI of a specific resources and together provided to a requesting consumer, the consumer will only have those access rights for the specified time that is specified by the SAS.

A BLOB Storage Shared Access Signature has the following 4 constraints:

• The Storage Resource: the type of resource (container, blob, table, queue)
• Start Time: the start time of SAS
• Expiry Time: how long is the SAS good for
• Permissions: the allowed operations (read, write, delete, list)

How It Works

Before getting into how to create one, let's start by breaking down a generated SAS into its components. Then we'll look at what is required for generating our own SAS, BLOB storage SAS limitations and what problems we need to be aware of.

Example of a URI with a Shared Access Signature:

https://MyAccount.blob.core.windows.net/demo/photo1.jpg?sv=2014-02-14&sr=b&st=2014-08-26T13%3A58%3A15Z&se=2014-08-26T13%3A58%3A25Z&sp=rw&sig=E0YB9qQBwtDGzX0KGgrrJf%2F2QM7cHgx7heoTtQ55Cwo%3D

blob URI	https://MyAccount.blob.core.windows.net/demo/photo1.jpg	The address to the BLOB resource.
sv	sv=2014-02-14	Storage Version (required): that specifies what version to use of the service.
sr	sr=b	Signed Resource (required): specifies what type of resource, in this case b for blob.
st	st=2014-08-26T13%3A58%3A15Z	Start Time (optional): ISO 8061 Format: The date time the SAS begins based on UTC time. Recommended to be avoided if possible due to problems with time synchronization of remote clients.
se	se=2014-08-26T13%3A58%3A25Z	Signed Expiry (required): ISO 8061 Format: The date the SAS expires based on UTC time when combined with the start date or implied when not specified, defines the valid time period.
sp	sp=rw	Signed Permissions: (required) The defined access control permissions for this resource. In this case, read and write permissions are granted
sig	sig=E0YB9qQBwtDGzX0KGgrrJf%2F2QM7cHgx7heoTtQ55Cwo%3D	Signature: The signature is an HMAC computed over the string-to-sign and key using the SHA256 algorithm and then encoded using Base64 encoding. This signature is what Azure uses to authenticate the provide SAS.

When a request to a BLOB storage resource comes in with an accompanied Shared Access Signature, the first thing that Azure will do is validate the authenticity of the supplied query parameters to ensure they have not been tampered with. This is done by computing the HMAC-SHA256 hash that is generated using the Azure Storage Access Key with the supplied parameters as the message and verify it matches the provided sig parameter. Upon success, it will then validate the start time (if provided) and that the SAS has not expired as well as the permissions. If all validations check out, the requestor will be granted access to the specified resource in the URI assuming the requested operation matches the supplied permissions.

Note: In order to take advantage of the security imposed by the Shared Access Signature, we need to limit/restrict the access given to the storage access resource whether that is a container or BLOB. In other words, don't set the Container's BlobContainerPublicAccessType to Container or BLOB or any consumer will be able to access a BLOB resource. See Shared Access Signatures Best Practices below for more information.

Generating a SAS

There is much more to be said about working with Shared Access Signatures and we'll get to that, but for now let's look at creating our own for use through the Storage Client SDK. We'll do two things here, generate a Shared Access Policy to specify the access rights and expiry time and generate the SAS by providing the policy:

In this specific example, we are generating a SAS on a specific BLOB resource, by creating a policy that specifies read and write only permissions with a start time of 1 day in the future, lasting for 10 minutes. This returned SAS string can then be appended to the BLOB URI that when supplied to a consumer can govern their access to the resource.

Tip: Don't specify a start time unless it is absolutely required due to problems with clients and time clock synchronization issues. This example wanted to provide the possible available policy settings. See Shared Access Signatures Best Practices for more information.

For Containers and BLOBs, the following are the available access rights that can be defined.

Containers:

• Read
• Write
• List
• Delete: note, this is allowing for any BLOB to be deleted. You can't provide access rights to delete the container.

BLOBs:

• Read
• Write
• Delete

This is a simple example of how to use the Storage Client SDK operations to generate a SAS. But there is much more to think about when using SAS such as how do we revoke a generated SDK and managing the lifetime of a SAS.

Managing Shared Access Signatures Lifetime and Revocation

Until now, we have been using what is called ad hoc Shared Access Signatures. This is when we define the policy at the time we generate the SAS. With ad hoc Shared Access Signatures, we have managed to provide access to BLOB storage resources without having to provide the Storage Access Keys.

You might have observed that the HMAC-SHA256 hashed signature (query parameter: sig) generated to validate the authenticity of the provided query parameters uses the Azure Storage Access Key. So, you might be asking, what if we decided that we need to invalidate a SAS that had been handed out? You guessed it, we would need to regenerate the Storage Access Key that was used. Which also means that anything else that relies on that Access Key would also be invalidated immediately and need updating. With the exception of regenerating the Storage Access Key, our only other means of managing the SAS is by specifying very short expiration times.

Tip: It is always recommended to use short expiration periods for any generated Shared Access Signatures. This is one of the handful of Best Practice tips.

To prevent this from having to occur, Azure has the concept of Stored Access Policies that the Shared Access Signatures are derived from when they are specified.

Stored Access Policies

In the previous ad hoc SAS, we generated the policy along with the SAS. Stored Access Policies are policies generated separate from any SAS and stored on the server. In regards to Azure Storage, it is the containers where the Stored Access Policies are associated and only a maximum of 5 policies can be stored at any given time. These policies can be viewed as templates that contain specific access rights and expiry times. Shared Access Signatures that are generated from specifying an existing policy, provide the following two noticeable benefits:

• Ease of revoking a generated Shared Access Signature without the need to regenerate Storage Access Keys
• Easily setup predefined access rights and expiry time periods

Once Stored Access Policies are in place, we can associate them with a Shared Access Signature by specifying the additional signedidentifier (si) query parameter:

si

si=policy1

(Optional) A unique value up to 64 characters in length that correlates to an access policy specified for the container.

We can add Stored Access Policies through the Storage Client SDK by setting the Permissions on a container that contains a collection of SharedAccessBlobPolicies as in the following:

We can then incorporate this policy into a SAS generation as in the following:
Notice that we have omitted the SharedAccessBlobPolicy since this is required when specifying a Stored Access Policy. Then, in the situation where we might need to revoke the generated Shared Access Signature, we only need to do one of the following to revoke any issued Shared Access Signature:

• Change the identifier (name) of the policy
• Remove the Stored Access Policy (in this case Policy1) from the container
• Provide a new list of Stored Access Policies that retains a policy with the same name but different parameters.

So, based on everything we have seen, when a consumer attempts to obtain a BLOB resource with a valid SAS they can proceed with operations that align with whatever rights have been granted by the SAS. But, in the case where a SAS is no longer valid due to expiration we can see an example of what this might look like attempting to acquire the resource.


Shared Access Signatures Best Practices

The following is a list of what I deemed the most important and security related Shared Access Signatures best practices documented by Microsoft Azure.

1. HTTPS Only! You might have noticed that in the creation of the Stored Access Policies or the Shared Access Signatures did we once mention an association with any client, user or application. This is because they allow any consumer that possess the Shared Access Signature to have access to the resource. Therefore, the #1 or at least top 3 most important best practice would be the importance of using HTTPS for any creation and communication of the SAS.
   
2. Adhere to the Principle of Least Privilege! This is only hinted at in the documentation, so I thought I would put it at the forefront. What I would consider probably the second most important security best practice for using SAS is, only issue the minimal credentials for the least amount of resources to a client that is required for them to do the necessary operations
   
3. Use Policies! If at all possible use Stored Access Policies. Policies allow for the creation of revocable Shared Access Signatures and avoid the undesirable need to regenerate Storage Access Keys as a way to revoke ad hoc Shared Access Signatures.
   
4. Issue short termed (near-term) policies! In either the case where you are issuing ad-hoc SAS or a revocable SAS has been compromised, policies with short term expirations minimize the exposed data's vulnerability.
   
5. Validate data written using SAS. When a client application writes data to your storage account, keep in mind that there can be problems with that data. If your application requires that that data be validated or authorized before it is ready to use, you should do this validation after the data is written and before it is used by your application. This practice also protects against corrupt or malicious data being written to your account, either by a user who properly acquired the SAS, or by a user exploiting a leaked SAS.
   
6. Don't always use SAS. Not all operations against your BLOB Storage account carry the same weight of importance. Therefore, don't always assume that a SAS is the perfect fit for all situations. If the risks of an operation outweighs any of the SAS benefits its time to rethink the situation. Use a middle tier of your application to do the necessary business rule validations, authentication, auditing and any further management processes. Finally, sometimes we tend to choose the harder path for no good reason, maybe a SAS is just adding unneeded overhead. Only you can judge the situation.

Managing/Access Tools

Finally, getting to the last part of what I feel is important to be aware of when working with Azure BLOB Storage is utilizing one of the handful of BLOB Storage Management tools. Some are free while others are not. These are not product reviews, but simply a way for you to be familiar with a few different BLOB Storage management tools that are available. MSDN blog site posted a Windows Azure Storage Explorers (2014) page for listing out a number of management tools available that you can also use for seeing what is available out in the wild!

Azure Storage Explorer

This is a free open-source application for viewing and performing various operations on your BLOB Storage resources. This will provide some of the basic operations such as read, delete and uploading and downloading of storage resources and creation of ad-hoc Stored Access Policies.



Azure Explorer

This is a very limited, free application by Cerebrata who provides Azure Explorer for performing some very rudimentary operations against your Azure Storage Account. Most of the limited operations include uploading, deleting, renaming and viewing containers and BLOBs. It is basically a teaser to their paid version Azure Management Studio.

The following you can see how Azure Explorer is displaying the hieratical structure like the BLOB we uploaded earlier using the “/” delimiter.



Azure Management Studio

Azure Management Studio and the Azure Explorer both are by Cerebrata. The Azure Management Studio is what Cerebrata rolled their paid version of Azure Explorer (Cloud Storage Studio) into. It comes with your full list of features for operating on your Azure Storage Account. These features include read, write, delete BLOB storage resources, creating and managing Stored Access Policies, managing properties and metadata on resources just to name a few.



Conclusion

We have covered a lot, from working with BLOB Containers to BLOB resources themselves to concurrency and asynchronous operations and a healthy dose of various security options that Azure provides out of the box. The intent was to provide more than just a quick how-to of Azure BLOB Storage, but cover some of the areas less talked about, understand some of the gotchas and provide tips that might help working with Azure BLOB Storage. That being said, it would take more than just a lengthy article to cover all the features and their variations. So, as you get underway with working with Azure BLOB storage, take advantage of their documentation for all the available features. Stay tuned for the next installment of “Be Sure with Azure” where we will look closely at a similar topic; Azure Table Storage.

References:

BLOB Storage http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/?rnd=1

Azure Subscriptions and Service Limits/Quotas http://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits/?rnd=1

Shared Access Signatures:

http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-shared-access-signature-part-1/

http://msdn.microsoft.com/en-us/library/azure/jj721951.aspx

Storage Client Repository: https://github.com/Azure/azure-storage-net

Tools: http://blogs.msdn.com/b/windowsazurestorage/archive/2014/03/11/windows-azure-storage-explorers-2014.aspx