Interacting with Azure Queues from PowerShell

In this post, I want to show how you can communicate between 2 (or more) machines with Azure Storage Queues using PowerShell. This type of communication naturally gives way to the competing consumer pattern, which is found throughout many high-scale systems. This pattern is quite useful for spreading work items across a pool of available compute resources in order to optimize throughput and availability.

Basic Architecture

A high-level diagram of the concept is show below:

Note that even though we are showing the senders and receivers as separate machines, this could just as easily be 2 processes on the same physical (or virtual) machine. This is one nice aspect of designing solutions with this pattern - we can easily transition to multi-machine deployments later as requirements change.

Why PowerShell?

While this example utilizes PowerShell, any language can take advantage of the techniques shown here - even those using the Azure Storage REST API directly. The original project I was working on already involved PowerShell scripts - your applications may have different requirements. Use what works for you!

Prerequisites

The following code is based on PowerShell v.5.1, along with the Azure PowerShell Cmdlets v.3.6.0 and Azure SDK v.2.9. You can easily install these items by using the Web Platform Installer, available for download here: https://www.microsoft.com/web/downloads/platform.aspx

The Code

The underlying code has two distinct parts: a client application that sends messages to a storage queue endpoint, and one or more receivers reading those messages from the queue and taking action (in this case, just printing a message).

The sender code (shown below) is fairly straightforward, sending some arbitrary messages in a loop. In this case, we are just sending a simple test string with a DateTime value appended to the end - just enough to see the different messages as the are received.

<#

DISCLAIMER: THIS SCRIPT IS OFFERED "AS IS" WITH NO WARRANTY. IT IS RECOMMENDED THAT YOU  
RUN IN A TEST ENVIRONMENT BEFORE USING IN YOUR PRODUCTION ENVIRONMENT.

#>

#
# Variables / consts
#
$storageAccountResourceGroup = "--Your Resource Group Name--"
$storageAccountName = "--Your Storage Account Name--"
$queueName = "--Your Queue Name--"

#
# Log message function
#
function LogMsg($msg){
    $now = [DateTime]::Now
    Write-Host "[$now]: $msg"
}

#
# Get primary storage key
#
LogMsg("Getting storage account key...")
$keys = Get-AzureRmStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroup `
            -Name $storageAccountName
$storageKey = $keys[0].Value

#
# Get storage account context
#
LogMsg("Getting storage context...")
$context = New-AzureStorageContext `
                -StorageAccountName $storageAccountName `
                -StorageAccountKey $storageKey

#
# Lookup queue to use
#
LogMsg("Locating message queue...")
$storageQueue = Get-AzureStorageQueue `
                    -Name $queueName `
                    -Context $context

#
# Create some sample messages (about 25)
#
LogMsg "Starting message send..."
for ($i = 0; $i -lt 25; $i++){
    $now = [DateTime]::Now.ToString()
    $payload = "This is just a test message sent at $now"

    $qmsg = New-Object Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage $payload
    $storageQueue.CloudQueue.AddMessage($qmsg)
    
    Write-Host "Sent message $i..."
}
LogMsg "Message send complete!"
    

On the receiving end, we see a very similar pattern: a while-loop that reads messages from the queue, or optionally sleeps for a bit if there are no messages available. However, notice in this case we have a simple exponential backoff built in to progressively sleep longer intervals when no messages are available. This avoids endless polling of the queue, and keeps the storage transactions to a minimum (note with Azure storage, you are charged for transactions).

<#

DISCLAIMER: THIS SCRIPT IS OFFERED "AS IS" WITH NO WARRANTY. IT IS RECOMMENDED THAT YOU  
RUN IN A TEST ENVIRONMENT BEFORE USING IN YOUR PRODUCTION ENVIRONMENT.

#>

#
# Variables / consts
#
$storageAccountResourceGroup = "--Your Resource Group Name--"
$storageAccountName = "--Your Storage Account Name--"
$queueName = "--Your Queue Name--"

#
# Log message function
#
function LogMsg($msg){
    $now = [DateTime]::Now
    Write-Host "[$now]: $msg"
}

#
# Get primary storage key
#
LogMsg("Getting storage account key...")
$keys = Get-AzureRmStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroup `
            -Name $storageAccountName
$storageKey = $keys[0].Value

#
# Get storage account context
#
LogMsg("Getting storage context...")
$context = New-AzureStorageContext `
                -StorageAccountName $storageAccountName `
                -StorageAccountKey $storageKey

#
# Lookup queue to use
#
LogMsg("Locating message queue...")
$storageQueue = Get-AzureStorageQueue `
                    -Name $queueName `
                    -Context $context

#
# Read queue messages, with a back-off on sleeps
#
$initSleep = 250
$sleepTime = $initSleep
LogMsg "Starting listener..."
while ($true){
    $qmsg = $storageQueue.CloudQueue.GetMessage($null)

    if ($qmsg -ne $null){
        $sleepTime = $initSleep
        Write-Host $qmsg.AsString
        $storageQueue.CloudQueue.DeleteMessage($qmsg)
    } else {
        # Apply an exponential-backoff sleep
        sleep -Milliseconds $sleepTime
        $sleepTime *= 2
        $sleepTime = [Math]::Min(5000, $sleepTime)
        LogMsg "Sleep interval now: $sleepTime"
    }
}
    

Putting it together

The screen capture below shows an example execution of these two scripts side-by-side. Notice we can see the receiver (on the right) pulling the messages off the queue and displaying them. Additionally, we can see the sleep intervals backing off when there are no more messages - this is the effect of the back-off code in the receiver sample above.

Leveraging queues for communication between disconnected parts of your application workflow is a powerful technique, and one that can be applied on the same machine or across multiple machines. Note however that in this example the messages are "one-way"; that is, we can only send a message to a receiver, but we can't get a direct response. In a later post, we'll discuss ways to get responses back to the sender, and look at some ways we can leverage this technique to build scalable, resilient systems.