Getting Started with Loosely Coupled Applications Using Azure Storage and Queues

Introduction

In this article, I'll explore a couple of ASP.NET and Azure concepts, including:

  1. The FileUpload control, and how to use it within Azure
  2. Synchronous uploading of data
  3. The requirement to further process our uploads, and,
  4. An Asynchronous pattern to do so using Azure Queues and Worker Roles

The FileUpload Control

Oftentimes, you'll want to upload some data to your web application. Much of the time, this is a file that the user has that your application needs. ASP.NET has long provided support for file uploading using the FileUpload control. It works exactly the same in Azure. So why are we talking about it? Well, the typical use case takes the file from the control, and saves it somewhere - typically to some persistent storage on the server. That's where the difference comes in. In Azure, the drives on your "servers" don't persist. They're not really yours to use.

So what do we do? Where will we keep our files? Blob storage to the rescue. It is persistent, and through it's access control, you can make it either private or public, which is often a requirement as well (you want the rest of the world to be able to see those uploaded pictures, right?)

Synchronous Uploading

So, let's get started. In our case, I'll focus on the uploading of image files, so we can build the next great photo sharing application. You can download the sample code here. You'll notice that the project already has two roles - one web role and one worker role. We'll start with the web role. Take a look at PictureUploadSynchronous.aspx in the PictureUploadAndProcessSampleWebSite project.

<body>
    <form id="form1" runat="server">
    <div>
		<asp:FileUpload ID="FileUpload1" runat="server" />
		<asp:Button ID="Button1" runat="server" Text="Upload" onclick="Button1_Click" />
		<asp:Label ID="StatusLabel" runat="server" />
    </div>
    </form>
</body>

If you've ever used the FileUpload control, this looks pretty typical. Generically, it supports the creation of a file selection dialog box (triggered by a button), and once your form is submitted, makes the data from that file available to the server for processing. In our case, we'll simply use the click event on Button1 to trigger that code. Let's take a look at that now. Open up PictureUploadSynchronous.aspx.cs and find the Button1_Click method.

protected void Button1_Click(object sender, EventArgs e)
{
	if (FileUpload1.HasFile)
	{
		try
		{
			// connect to account, client and container
			CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
			CloudBlobClient blobStorage = storageAccount.CreateCloudBlobClient();
			CloudBlobContainer container = blobStorage.GetContainerReference("uploadedpics");
			container.CreateIfNotExist();

			// get the handle to the blob
			string blobName = FileUpload1.FileName;
			var blob = container.GetBlockBlobReference(blobName);
			blob.Properties.ContentType = FileUpload1.PostedFile.ContentType;

			// upload our file
			blob.UploadFromStream(FileUpload1.FileContent);

			StatusLabel.Text = "Upload status: File uploaded!";
		}
		catch (Exception ex)
		{
			StatusLabel.Text = "Upload status: The file could not be uploaded. The following error occured: " + ex.Message;
		}
	}
}

The edges of this look very similar to the use of the control in a non-Azure environment, but the chewy center has changed a bit. 

if (FileUpload1.HasFile)
{
...
}

This simply determines if the FileUpload control has been told of a file to upload. Since it's possible that the user simply clicked on Button1 before they chose a file, we do this sanity check.

// connect to account, client and container
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
CloudBlobClient blobStorage = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobStorage.GetContainerReference("uploadedpics");
container.CreateIfNotExist();

Here we connect to the persistent blob storage account and use the factory to get a CloudBlobClient up and running. Once we have that, we can get a handle to the particular container that we'll use as a location for our persisted files.

// get the handle to the blob
string blobName = FileUpload1.FileName;
var blob = container.GetBlockBlobReference(blobName);
blob.Properties.ContentType = FileUpload1.PostedFile.ContentType;

Here, we get a reference to a new blob in preparation for uploading our file into this blob. Note that if the blob already exists, our upload will just overwrite the previous contents. If it doesn't already exist, it will be created automatically during the upload. Also, notice that we just reuse the filename of the uploaded file as the name of the blob. Finally, we'll set the ContentType of the blob to be the same as the ContentType of the uploaded file. This ensures that when we later want to download and look at our pretty pictures, ContentType handling in the browser will work as expected.

// upload our file
blob.UploadFromStream(FileUpload1.FileContent);

Finally, we just upload the file stream available from the FileUpload control (in the FileContent property) as the stream to put into the blob. If that's all we wanted to do, we'd be pretty much done now. We've uploaded the file and put it into persistent storage so that we have it available when we want it later. But imagine if we wanted to really do something with those images. Most photo sharing sites offer thumbnails of the images to browse before downloading the larger images. Many offer multiple thumbnail sizes as well. Let's do the same thing in our site.

Thumbnailing

As it turns out, .NET provides support for the resizing of images in the System.Drawing namespace. We can take easy advantage of that to do our thumbnailing. We already learned how to create blobs, so why don't we just add some code to the Button1_Click event to do the thumbnailing and create the new blobs?

Well, we could do that, but it's not ideal. For starters, it would cause a lot more processing on the server side during the page lifecycle, and delay the response back to the user, causing the UI to be sluggish - never a good thing. Additionally, if we had any problems at all during the thumbnailing, the best we could do is throw back an error to the user, potentially after a long amount of processing time. There must be a better way, right?

As it turns out, this is a perfect time to set up a background process to do the heavy lifting. Let's take a look at how we can do this.

Queues and Worker Roles

In the non-Azure world, this would have called for a background processing methodology as well. We might well have coded it up using MSMQ, or maybe rolled our own queue, writing data into a database and then with a Windows Service polling the table occasionally to do the work. Alternatively, we could have set up a FileSystemWatcher object to poll the upload directory looking for new files, and do our work when we saw one.

In Azure, however, the concept of a Queue is a first class citizen, and we should just go ahead and take advantage of it. That's half the battle. Once we have our work saved in a queue message, we need something else to pick it up and get it done. That's what we'll use our Worker Role for.

Let's take a look at PictureUploadAsynchronous.aspx. 

<body>
    <form id="form1" runat="server">
    <div>
		<asp:FileUpload ID="FileUpload1" runat="server" />
		<asp:Button ID="Button1" runat="server" Text="Upload" onclick="Button1_Click" />
		<asp:Label ID="StatusLabel" runat="server" />
    </div>
    </form>
</body>

Wait, that looks exactly like the other one, right? Well, yes. It does. Nothing changed in the markup. We'll start to see the rest of the functionality in the code-behind. Open up PictureUploadAsynchronous.aspx.cs and check out the Button1_Click method now.

protected void Button1_Click(object sender, EventArgs e)
{
	if (FileUpload1.HasFile)
	{
		try
		{
			// connect to account, client and container
			CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
			CloudBlobClient blobStorage = storageAccount.CreateCloudBlobClient();
			CloudBlobContainer container = blobStorage.GetContainerReference("uploadedpics");
			container.CreateIfNotExist();

			// get the handle to the blob
			string blobName = FileUpload1.FileName;
			var blob = container.GetBlockBlobReference(blobName);
			blob.Properties.ContentType = FileUpload1.PostedFile.ContentType;

			// upload our file
			blob.UploadFromStream(FileUpload1.FileContent);
			StatusLabel.Text = "Upload status: File uploaded!";

			// ok, it's good that we uploaded the file, but now we need to process it into a thumbnail, and other sizes.
			CloudQueueClient cloudQueueClient = storageAccount.CreateCloudQueueClient();
			CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("uploadedpictures");
			cloudQueue.CreateIfNotExist();
			CloudQueueMessage cloudQueueMessage = new CloudQueueMessage(blobName);
			cloudQueue.AddMessage(cloudQueueMessage);
		}
		catch (Exception ex)
		{
			StatusLabel.Text = "Upload status: The file could not be uploaded. The following error occured: " + ex.Message;
		}
	}
}

The first part of this looks exactly the same, and it is. Nothing has changed as far as the upload of the file goes. We still do the upload, and put it into a blob. What has changed is what we do after the file is uploaded.

// ok, it's good that we uploaded the file, but now we need to process it into a thumbnail, and other sizes.
CloudQueueClient cloudQueueClient = storageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("uploadedpictures");
cloudQueue.CreateIfNotExist();
CloudQueueMessage cloudQueueMessage = new CloudQueueMessage(blobName);
cloudQueue.AddMessage(cloudQueueMessage);

Here, we use the factory to create a CloudQueueClient, similar to the CloudBlobClient. Notice that the storage account handles both of these operations. The CloudStorageAccount has Blob, Queue and Table capabilities, although we won't discuss the Table capabilities here, as we don't need them for this sample.

Using the CloudQueueClient, we'll get a handle to the CloudQueue called "uploadedpictures", and create it if it doesn't already exist. You can call this whatever you'd like, but it's important that it only contains lower case letters, no upper case. Finally, we'll add a new CloudQueueMessage to the queue. In our case, we'll simply use the filename to process as the message.

OK, great! Our file is uploaded, and we're good to go, right? Yep, and there's only one final step. We need to do something with that queue message and get our thumbnailing done. Let's create a WorkerRole to accomplish that.

The Workhorse - Worker Role

The worker role's job is clear. Dequeue messages from the "uploadedpictures" queue, and create the thumbnails. Let's see how we accomplish that. Open up WorkerRole.cs in the PictureUploadAndProcessSampleWorkerRole project and look at the Run method. This is automatically invoked when our Worker Role starts up.

public override void Run()
{
	try
	{
		// grab the reference to the queue we work against...
		CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
		CloudQueueClient cloudQueueClient = storageAccount.CreateCloudQueueClient();
		CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("uploadedpictures");
		cloudQueue.CreateIfNotExist();

		while (true)
		{
			// grab up to 5 messages for this pass...
			var messages = cloudQueue.GetMessages(5);
			foreach (CloudQueueMessage message in messages)
			{
				string pathToHandle = message.AsString;
				CreateThumbnail(pathToHandle, 100, 100);
				cloudQueue.DeleteMessage(message);
			}

			Thread.Sleep(30000);
		}
	}
	catch (Exception ex)
	{
		string s = ex.ToString();
	}
}

The pattern here is pretty straightforward. First we do whatever startup activities we need to, and then we spin, look for work, and take a breath. In our case, the startup activity is to get a handle to the queue we're watching.

// grab the reference to the queue we work against...
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
CloudQueueClient cloudQueueClient = storageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("uploadedpictures");
cloudQueue.CreateIfNotExist();

This is exactly the same as the behavior in the web page. The CreateIfNotExist call might seem a little odd here, but in the wild, we don't know if the web role or the worker role will get to the queue first, so we just defensively make sure the queue exists no matter who wins that race. Next, the spin, work, breathe cycle for us looks like this:

while (true)
{
	// grab up to 5 messages for this pass...
	var messages = cloudQueue.GetMessages(5);
	foreach (CloudQueueMessage message in messages)
	{
		string pathToHandle = message.AsString;
		CreateThumbnail(pathToHandle, 100, 100);
		cloudQueue.DeleteMessage(message);
	}

	Thread.Sleep(30000);
}

Here, we simply ask the queue for up to 5 messages to process, and if we have any, we take the path out of the message and pass it on to our CreateThumbnail method to process.

public void CreateThumbnail(string file, int width, int maxHeight)
{
	// connect to account, client and container
	CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageAccountConnectionString"]);
	CloudBlobClient blobStorage = storageAccount.CreateCloudBlobClient();
	CloudBlobContainer container = blobStorage.GetContainerReference("uploadedpics");
	container.CreateIfNotExist();

	// get the blob into a stream
	string newFile = "thumb" + file;
	var blob = container.GetBlockBlobReference(file);
	System.IO.MemoryStream blobStream = new System.IO.MemoryStream();
	blob.DownloadToStream(blobStream);

	// create an image from the stream...
	System.Drawing.Image blobImage = System.Drawing.Image.FromStream(blobStream);
		
	if (blobImage.Width <= width)
		width = blobImage.Width;

	int height = blobImage.Height * width / blobImage.Width;
	if (height > maxHeight)
	{
		width = blobImage.Width * maxHeight / blobImage.Height;
		height = maxHeight;
	}

	// build the thumbnail
	System.Drawing.Image thumbFile = blobImage.GetThumbnailImage(width, height, null, IntPtr.Zero);

	// save out our thumbnail...
	var thumbnailBlob = container.GetBlockBlobReference(newFile);
	thumbnailBlob.Properties.ContentType = blob.Properties.ContentType;

	// upload our file
	System.IO.MemoryStream thumbnailStream = new System.IO.MemoryStream();
	thumbFile.Save(thumbnailStream, blobImage.RawFormat);
	// VERY IMPORTANT - SET STREAM POSITION BACK TO ZERO!
	thumbnailStream.Position = 0;
	thumbnailBlob.UploadFromStream(thumbnailStream);
	blobImage.Dispose();
	thumbFile.Dispose();
}

I won't spend much time describing this method here, as it's not really Azure specific. Also note that the code at the beginning of the method to grab the CloudStorageAccount and the CloudBlobContainer could certainly be cached and not run redundantly, but it would complicate the code here. Feel free to make that change in your implementation.

Summary

Handling long running processes in the background, and keeping the user experience moving is a best practice. The use of CloudQueues and Worker Roles in Azure helps us to accomplish this goal.

blogroll

social